Data processing method and device for AI chip, chip and computer equipment

Abstract

Embodiments of the present application disclose a data processing method and device for an AI chip, a chip and a computer device, and belong to the technical field of chips. After obtaining the structural features of a neural network, at least two candidate inter-layer fusion strategies can be determined based on the features, and a target inter-layer fusion strategy that meets the least target exchange amount and the peak value of inference memory consumption that meets the total on-chip cache capacity of the AI chip is selected from the at least two candidate inter-layer fusion strategies. Finally, the adjacent network layers in the neural network are fused based on the target inter-layer fusion strategy to obtain a fused neural network, so that the to-be-reasoned data is processed based on the fused neural network. Since the target exchange amount is the data exchange amount between the on-chip cache and the off-chip storage in the target dimension in a single inference, the present application can determine the available target inter-layer fusion strategy to fuse the neural network while reducing the on-chip and off-chip data exchange amount, thereby improving the inference efficiency of the neural network while the AI chip hardware remains unchanged.

Description

Technical Field

[0001] This application relates to the field of chip technology, and in particular to a data processing method, apparatus, chip, and computer device for AI chips. Background Technology

[0002] Artificial intelligence (AI) chips, also known as AI accelerators, are a type of hardware accelerator specifically designed to accelerate artificial intelligence algorithms.

[0003] In related technologies, the inference process of running neural networks on AI chips is typically limited by the capacity of the AI ​​chip's on-chip cache (Static Random-Access Memory, SRAM). Therefore, AI chips usually divide the input feature maps of the neural network layers into blocks, read them into the on-chip cache, process the data on the AI ​​chip, and then output the data from the neural network's output layer to off-chip storage (Double Data Rate SDRAM, DDR). This process is repeated until each block of input feature maps has been processed. The output data obtained from the off-chip storage is then concatenated to form the final result data.

[0004] However, the above processing method is greatly limited by the bandwidth of off-chip storage and requires frequent off-chip access operations, which affects the processing efficiency of AI chips for neural networks. Summary of the Invention

[0005] This application provides a data processing method, apparatus, chip, and computer device for AI chips. The technical solution is as follows:

[0006] According to one aspect of this application, a data processing method for an artificial intelligence (AI) chip is provided, the method comprising:

[0007] Obtain the structural features of the neural network;

[0008] Based on the structural features, at least two candidate layer fusion strategies are determined, which are used to fuse the network layers in the neural network.

[0009] From at least two candidate inter-layer fusion strategies, a target inter-layer fusion strategy is determined. The target exchange volume and peak inference memory consumption of the neural network after fusion using the target inter-layer fusion strategy meet preset conditions. The target exchange volume is used to indicate the amount of data exchanged between the on-chip cache and off-chip storage of the AI ​​chip in the target dimension during a single inference process. The peak inference memory consumption is used to indicate the maximum value of the on-chip cache consumed by the neural network during inference.

[0010] Based on the target inter-layer fusion strategy, adjacent network layers in the neural network are fused to obtain a fused neural network;

[0011] Based on the fused neural network, the data to be inferred is processed.

[0012] According to another aspect of this application, a data processing apparatus for an AI chip is provided, the apparatus comprising:

[0013] The feature acquisition module is used to acquire the structural features of the neural network;

[0014] The first determining module is used to determine at least two candidate inter-layer fusion strategies based on the structural features, wherein the candidate inter-layer fusion strategies are used to fuse the network layers in the neural network;

[0015] The second determining module is used to determine a target inter-layer fusion strategy from at least two candidate inter-layer fusion strategies. The target exchange volume and the peak inference memory consumption of the neural network after fusion using the target inter-layer fusion strategy meet preset conditions. The target exchange volume is used to indicate the amount of data exchanged between the on-chip cache and off-chip storage of the AI ​​chip in the target dimension during a single inference process. The peak inference memory consumption is used to indicate the maximum value of the on-chip cache consumed by the neural network during the inference process.

[0016] The network fusion module is used to fuse adjacent network layers in the neural network based on the target inter-layer fusion strategy to obtain a fused neural network.

[0017] The data processing module is used to process the data to be inferred based on the fused neural network.

[0018] According to another aspect of this application, an AI chip is provided, which includes a data processing device as provided in this application.

[0019] According to another aspect of this application, a computer device is provided, which includes a central processing unit (CPU) chip, an AI chip, and a memory, wherein the CPU chip, the AI ​​chip, and the memory are connected via a bus, and the AI ​​chip includes a data processing device as provided in this application.

[0020] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the instruction being loaded and executed by a processor to implement the data processing method for an AI chip as provided in various aspects of this application.

[0021] According to one aspect of this application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method for an AI chip provided in the various alternative implementations described above.

[0022] The beneficial effects of the technical solutions provided in this application embodiment may include:

[0023] After acquiring the structural features of the neural network, this application can determine at least two candidate inter-layer fusion strategies based on these features, select a target inter-layer fusion strategy from them, and finally fuse adjacent network layers in the neural network based on the target inter-layer fusion strategy to obtain a fused neural network. This fused neural network is then used to process the data to be inferred. Since the target exchange volume and peak inference memory consumption of the neural network fused using the target inter-layer fusion strategy meet preset conditions during inference, and the target exchange volume indicates the amount of data exchanged by the AI ​​chip's on-chip cache in the target dimension during a single inference process, this application can determine an available target inter-layer fusion strategy to fuse the neural network while reducing on-chip and off-chip data exchange volume. This improves the inference efficiency of the neural network without changing the AI ​​chip hardware and the neural network itself. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the structure of a computer device involved in this application;

[0026] Figure 2This is a schematic diagram illustrating the application of an interlayer fusion method in related technologies;

[0027] Figure 3 This is a schematic diagram illustrating the application of a data flow method in related technologies;

[0028] Figure 4 This is a schematic diagram of a boundary-preserving operation in related technologies;

[0029] Figure 5 This is a flowchart illustrating a data processing method for an AI chip, provided in an exemplary embodiment of this application.

[0030] Figure 6 This is a flowchart of a data processing method for an AI chip provided by another exemplary embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the hardware execution logic of a fused neural network provided in this application;

[0032] Figure 8 This is a schematic diagram of a tree-structured processing logic provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of a tree-structured data processing flow provided in an embodiment of this application;

[0034] Figure 10 This is a structural block diagram of a data processing apparatus for an AI chip provided in an exemplary embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, "connected" and "linked" can mean a fixed connection, a detachable connection, or an integral connection; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" is used to describe the relationship between related objects, indicating that three types of relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0038] As used herein, the term “if” may optionally be interpreted, depending on the context, as “when,” “in the event of,” “in response to determination,” or “in response to detection.” Similarly, depending on the context, the phrases “if it is determined that…” or “if (the stated condition or event) is detected” or “in response to the detection of (the stated condition or event).”

[0039] It should be noted that the use of personally identifiable information should comply with generally accepted industry or governmental privacy policies and practices that meet or exceed the requirements for protecting user privacy. Specifically, the nature of authorized use of personally identifiable information should be clearly explained to users during its management and processing to minimize the risk of unintentional or unauthorized access or use.

[0040] With the continuous development and widespread application of AI technology, the functions of AI networks are becoming increasingly powerful. Correspondingly, the structure of AI networks is becoming more complex, and the computational load when using AI networks for inference is also increasing. Therefore, the processing speed requirements for AI networks in computer devices are also rising. AI chips are hardware components developed to meet this demand.

[0041] For illustrative purposes, please refer to Figure 1 , Figure 1This is a schematic diagram of a computer device according to this application. The computer device 100 includes a CPU chip 110, an AI chip 120, and off-chip storage 130. Optionally, the computer device 100 may also include a DMA (Direct Memory Access) controller 140. The DMA controller 140 is used to control data exchange between hardware devices at different speeds. The devices can also exchange data via a bus 150.

[0042] In computer device 100, CPU chip 110 controls AI chip 120 to acquire a neural network. After acquiring the structural features of the neural network, AI chip 120 can determine at least two candidate inter-layer fusion strategies based on these features, and then determine a target inter-layer fusion strategy from these two candidate strategies. This allows AI chip 120 to fuse adjacent network layers in the neural network based on the target inter-layer fusion strategy, resulting in a fused neural network. AI chip 120 then processes the data to be inferred through the fused neural network, thereby realizing the operation of processing data to be inferred based on the neural network.

[0043] In one possible approach, the AI ​​chip 120 may perform the above process only during the initialization of the neural network. After the neural network is initialized, the AI ​​chip stores the neural network's identifier and the corresponding target inter-layer fusion strategy in off-chip memory 130. When the CPU chip 110 needs to reschedule the neural network for inference of data to be inferred, the CPU chip 110 obtains the corresponding target inter-layer fusion strategy based on the neural network's identifier. Subsequently, the AI ​​chip 120 fuses adjacent network layers in the neural network according to the target inter-layer fusion strategy to obtain a fused neural network. The AI ​​chip 120 then uses the fused neural network to process the data to be inferred.

[0044] In another possible approach, if the AI ​​chip 120 does not have a mechanism to store the identifier of the neural network and the corresponding target inter-layer fusion strategy in the off-chip storage 130, then the CPU chip 110 needs to perform the above process of first determining the candidate inter-layer fusion strategy and then determining the target inter-layer fusion strategy from the candidate inter-layer fusion strategies before each scheduling of the neural network.

[0045] It should be noted that when the CPU chip 110 needs to call the neural network to process the data to be inferred in the external storage 130, the CPU chip 110 instructs the AI ​​chip 120 to read the data to be inferred from the external storage 130. When a layer of the fused neural network completes its calculation, the AI ​​chip 120 transfers the intermediate data output by that layer to the external storage 130, and then reads the intermediate data back from the external storage 130 to the next layer of the neural network for calculation, until the output layer of the neural network obtains the data.

[0046] For example, if a neural network includes an input layer, a first convolutional layer (conv1), a second convolutional layer (conv2), a third convolutional layer (conv3), and an output layer, then depending on different inter-layer fusion strategies, the neural network can be fused into a combination of different blocks.

[0047] When the number of layers in the resulting neural network after fusion is 1, conv1, conv2, and conv3 are fused into a single layer, meaning this layer is obtained by fusing three data layers. In the fused neural network, since the three data layers are fused into one layer, the input to this layer is the input layer data, and the output is the output layer data. That is, there is no intermediate data in the fused neural network. After the AI ​​chip 120 reads inference data from the external storage 130, the data it outputs to the external storage 130 is the result data obtained after neural network inference.

[0048] When the number of layers in the fused neural network is 2, conv1, conv2, and conv3 are fused into two layers. In one possible approach, conv1 and conv2 are fused into one layer, and conv3 is another layer. In this case, after the AI ​​chip 120 reads inference data from external storage 130, it outputs the intermediate data obtained after conv2 convolution to external storage 130, then reads this intermediate data from external storage 130 into conv3, where it performs subsequent calculations until the output layer of the neural network outputs the result data. In another possible approach, conv1 is a single layer, and conv2 and conv3 are fused into another layer. In this case, after the AI ​​chip 120 reads inference data from external storage 130, it outputs the intermediate data after conv1 convolution to external storage 130, then reads this intermediate data from external storage 130 into the layer formed by fused conv2 and conv3, where it performs subsequent calculations until the output layer of the neural network outputs the result data.

[0049] It should be noted that if the target dimension is rows and the resulting data consists of 12 rows, and the neural network can obtain 6 rows of data in one computation, then the neural network needs to perform inference data twice to obtain the 12 rows of resulting data. Alternatively, the target dimension could also be columns.

[0050] In this application, deep neural networks (DNNs) typically contain a large number of network layers. On-chip caches in AI chips generally offer higher access performance than off-chip storage, and data transfer bandwidth is limited. To improve the efficiency of data processing in AI chips, this application caches data in an on-chip cache, reducing data exchange between the on-chip cache and off-chip storage. This design concept is one of the design principles of this application.

[0051] The following describes two methods commonly used in this field to reduce data exchange between on-chip cache and off-chip storage.

[0052] The first method is layer fusion. This method merges multiple network layers in a neural network, maximizing the amount of computation the AI ​​chip can perform simultaneously while minimizing data exchange between on-chip cache and off-chip storage. This method prioritizes breadth over depth. Since on-chip cache space is limited, it's generally impossible to load inference data into the on-chip cache all at once. Therefore, the AI ​​chip will segment the inference data. The segmentation can be done along either row or column dimensions.

[0053] Optionally, the inference data can be a feature map.

[0054] It should be noted that the inference data in this application generally includes four dimensions: NHWC (layout of feature maps, where N refers to the number dimension, H refers to the height dimension, W refers to the width dimension, and C refers to the channel dimension). AI chips can divide the inference data into several small blocks of the same size from the height dimension (i.e., the row dimension) or from the width dimension (i.e., the column dimension).

[0055] To illustrate, an AI chip can not only divide inference data into several small blocks of the same size from one target dimension, but also from two target dimensions simultaneously. For example, an AI chip can divide inference data into several small blocks of the same size from both the height dimension (i.e., the row dimension) and the width dimension (i.e., the column dimension).

[0056] Because data segmentation during inference leads to redundant computations in the neural network, it reduces the inference efficiency of the neural network. Furthermore, the amount of redundant computation in inter-layer fusion methods increases with the number of fuselayers, or conversely, with the number of segments. Therefore, the number of layers and the number of segments in inter-layer fusion methods are limited in practical applications.

[0057] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the application of an inter-layer fusion method in related technologies. Figure 2 In the neural network, there are an input layer 210, a first convolutional layer 220, a second convolutional layer 230, a third convolutional layer 240, and an output layer 250.

[0058] exist Figure 2 In this model, the first convolutional layer 220, the second convolutional layer 230, and the third convolutional layer 240 are merged into a single layer block. The kernel of each convolutional layer has a dimension of 3, meaning the kernel is 3x3 in size. In this scenario, the target dimension can be defined as the number of rows, meaning that 3 rows of input data result in 1 row of output data.

[0059] Assuming the input data is split into two parts by rows, the first part of the data is input by input layer 210 in 12 rows (i.e., Figure 2 The first convolutional layer (220) outputs 10 rows of data (represented by line segment AB plus the 6 rows of data represented by line segment BC). Figure 2 The middle line segment EF represents 6 rows of data, plus the line segment FG represents 4 rows of data. The second convolutional layer 230 outputs 8 rows. Figure 2 The middle line segment JK represents 6 rows of data, plus the line segment KL represents 2 rows of data. The third convolutional layer 240 outputs 6 rows, which is exactly half of the required result data. The second set of data is input by the input layer 210 with 12 rows ( Figure 2 The data consists of 6 rows represented by line segment BC plus 6 rows represented by line segment CD, with 6 rows (represented by line segment BC) overlapping with the first data set. The first convolutional layer 220 outputs 10 rows. Figure 2 The data consists of 4 rows represented by line segment FG and 6 rows represented by line segment GH, with 4 rows (represented by line segment FG) overlapping with the first data set. The second convolutional layer 230 outputs 8 rows (...). Figure 2 The first convolutional layer (240) outputs 6 rows of data (represented by line segment KL and 6 rows of data represented by line segment LM), with 2 rows overlapping with the first data set. This outputs exactly half of the required result data. Finally, the output layer (250) merges the 6 rows of data obtained from the first data set with the 6 rows of data obtained from the second data set to produce the final output.

[0060] When calculating the data in the first convolutional layer 220, the overlapping portion of its input causes 6 rows of redundant calculations. When calculating the data in the second convolutional layer 230, the overlapping portion of its input causes 4 rows of redundant calculations. When calculating the data in the third convolutional layer 240, the overlapping portion of its input causes 2 rows of redundant calculations. It can be seen that the amount of redundant calculations increases with the number of layers in the fused block and with the number of segments.

[0061] against Figure 2 The method shown reduces data exchange between on-chip cache and off-chip storage. Assume the on-chip cache only has enough space to store 12 lines of input data and 10 lines of output data. If, after calculating the first data in the first convolutional layer 220, the second convolutional layer 230 is not started to continue calculating the first data, but instead the second data in the first convolutional layer 220 is calculated (during this calculation, the AI ​​chip does not need to calculate the overlapping part, only the last 6 lines of the second data), then the 10 lines of output from the first data must be temporarily stored in off-chip storage. When the second convolutional layer 230 needs to be calculated, the 10 lines of output from the first data are reloaded into the on-chip cache, thus creating data exchange between on-chip cache and off-chip storage. As the number of layers in the neural network increases, the inter-layer fusion method cannot completely avoid data exchange between on-chip cache and off-chip storage, and the proportion of redundant calculations also increases sharply. The inter-layer fusion method usually strikes a trade-off between the two.

[0062] For 2D data to be reasoned, the inter-layer fusion method can also split the row and column dimensions simultaneously to achieve the processing of 2D data to be reasoned.

[0063] The second method is the data streaming approach. When AI chips use the data streaming approach, they typically read one line of data to be inferred from off-chip memory at a time, calculating as many network layers as possible simultaneously. In other words, the data streaming approach follows a depth-first principle. The data streaming approach can largely avoid data exchange between on-chip cache and off-chip memory. However, after calculating each network layer, the data streaming approach requires a logical check to determine whether the conditions for calculating the next network layer are met. Therefore, the data streaming approach requires repeated logical checks, placing high demands on hardware performance.

[0064] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the application of a data flow method in related technologies. Figure 3 In this neural network, there are an input layer, a first convolutional layer 310, a second convolutional layer 320, a third convolutional layer 330, and an output layer.

[0065] exist Figure 3 In the process, when the input layer reads the first line of inference data, the AI ​​chip performs a logical judgment and determines that it is not possible to call the first convolutional layer 310 for calculation based on the first line of inference data.

[0066] In this situation, the AI ​​chip continues to read the second line of inference data, and then performs a logical judgment. It is determined that it is still unable to call the first convolutional layer 310 for calculation based on the above two lines of inference data.

[0067] In this situation, the AI ​​chip continues to input the third line of inference data. The AI ​​chip then performs a logical judgment. If it determines that the calculation can be performed using the first convolutional layer 310 based on the above three lines of inference data, it calls the first convolutional layer 310 to calculate the above three lines of inference data and outputs the first line of output data from the first convolutional layer 310. If it determines that the calculation cannot be performed using the second convolutional layer 320, then...

[0068] In this situation, the AI ​​chip continues to input the fourth line of inference data. If it determines that calculations can be performed based on the second to fourth lines of inference data, it calls the first convolutional layer 310 to calculate the three lines of inference data and outputs the second line of output data from the first convolutional layer 310. If it determines that calculations cannot be performed by calling the second convolutional layer 320, it will be determined that the second convolutional layer 320 cannot be called.

[0069] In this situation, the AI ​​chip continues to input the 5th line of inference data, determines that the first convolutional layer 310 can be called for calculation, and outputs the 3rd line of output data from the first convolutional layer 310. It then determines that the second convolutional layer 320 can be called for calculation, and outputs the 1st line of output data from the second convolutional layer 320. Finally, it determines that the third convolutional layer 330 cannot be called for calculation.

[0070] In this situation, the AI ​​chip continues to input the 6th line of inference data, determines that the first convolutional layer 310 can be calculated, and outputs the 4th line of output data from the first convolutional layer 310. It determines that the second convolutional layer 320 can be called for calculation, and outputs the 2nd line of output data from the second convolutional layer 320. It determines that the third convolutional layer 330 cannot be called for calculation, and continues to input...

[0071] In practice, due to operations such as pooling, convolution (with a stride of 2), resize-bilinear (image scaling and bilinear interpolation), concat operations, and shortcuts, the logical judgments and operation jumps performed using the dataflow method become more complex. Therefore, while the dataflow method can largely avoid data exchange between on-chip cache and off-chip storage, it requires a logical judgment and operation jump for each network layer computed, placing high demands on hardware performance.

[0072] It should be noted that the dataflow method can also handle 2D inference data, but this will not be discussed further.

[0073] It should be noted that the dataflow method generally employs a ring buffer mechanism in its implementation. This mechanism preserves the boundary calculation results of intermediate network layers, thus completely avoiding redundant computation. In practice, the inter-layer fusion method also employs two methods in its implementation: recompute and reuse. Recompute performs redundant computations, while reuse completely avoids redundant computations by preserving the boundary calculation results of intermediate network layers.

[0074] Please see Figure 4 , Figure 4 This is a schematic diagram of a boundary-preserving operation in related technologies. Figure 4 In this paper, taking the inter-layer fusion reuse operation method as an example, we introduce a method to retain the boundary calculation results of intermediate network layers.

[0075] Similarly, assuming the AI ​​chip divides the inference data into two parts by row, the first part of the data is input into the input layer 410 with 12 rows, and the two boundary data (i.e., ...) in the input layer 410 are then divided into two parts. Figure 4 The data represented by the line segment NC is stored in the on-chip cache. The first convolutional layer 420 outputs 10 lines (that is... Figure 4 The data corresponding to the middle line segment EG), and the two rows of boundary data in the first convolutional layer 420 (that is... Figure 4 The data represented by the midline segment PG is stored in the on-chip cache, and the second convolutional layer 430 outputs 8 rows (that is... Figure 4 The data corresponding to the middle line segment JL), and the two rows of boundary data in the second convolutional layer 430 (that is... Figure 4 The data represented by the middle line segment KL is stored in the on-chip cache. The third convolutional layer 440 outputs 6 lines, which is exactly half of the required result data.

[0076] The second set of data to be reasoned is input into the input layer 410 for 6 rows (that is...) Figure 4 The data represented by the middle line segment CD) reuses the two rows of boundary data from the retained input layer 410 (i.e. Figure 4 The data represented by the middle line segment NC), the first convolutional layer 420 outputs 6 rows (that is... Figure 4 The data represented by the midline segment GH is reused from the two rows of boundary data of the retained first convolutional layer 420 (i.e., Figure 4 The data represented by the midline segment PG), the second convolutional layer 430 outputs 6 rows (that is... Figure 4 The data represented by the midline segment LM is reused from the two rows of boundary data of the reserved second convolutional layer 430 (i.e. Figure 4The data represented by the line segment KL is output by the third convolutional layer 440, which outputs 6 rows, exactly half of the required result data. As you can see, overlap! = the number of rows to retain. For example, the boundary data of the first and second data sets has a total of 6 rows, but only the 2 rows of the shaded area need to be retained. This completely avoids redundant calculations while adding a small amount of on-chip cache.

[0077] It should be noted that the breadth-first search algorithm is typically implemented using exhaustive search or dynamic programming to search for the optimal partitioning method along the h and w dimensions. The time complexity of these methods is generally K·n. 2 The order of magnitude is , where K represents the number of feasible hw splitting methods and n represents the number of network layers.

[0078] This illustration demonstrates the hardware execution logic of the fused neural network after batch input of inference data provided in this application. This execution logic includes two processing stages: Stage 1 and Stage 2.

[0079] Phase One. The first set of data input into the neural network for inference does not contain pre-reserved boundary data. Therefore, the computational logic of the AI ​​chip differs from Phase Two. The specific process is as follows: The first set of input data consists of 12 lines. The AI ​​chip stores the last two lines of the first set of data as boundary data in an on-chip cache. The first convolutional layer outputs 10 lines, and the last two lines of the first convolutional layer are also stored as boundary data in the on-chip cache. The second convolutional layer outputs 8 lines, and the last two lines of the second convolutional layer are also stored as boundary data in the on-chip cache. The third convolutional layer, as the last network layer used for computation, outputs 6 lines of data as part of the final result data.

[0080] Phase Two. The second / third / and subsequent data sets require the use of pre-reserved boundary data. The specific process is as follows: The second set of inference data input consists of 6 lines. The AI ​​chip reuses the 2 lines of boundary data corresponding to the reserved input layer and stores the last 2 lines of this input inference data as boundary data in the on-chip cache. The first convolutional layer outputs 6 lines, reuses the 2 lines of boundary data corresponding to the reserved first convolutional layer, and stores the last 2 lines of the output data of the first convolutional layer as boundary data in the on-chip cache. The second convolutional layer outputs 6 lines, reuses the 2 lines of boundary data corresponding to the reserved second convolutional layer, and stores the last 2 lines of the output data of the second convolutional layer as boundary data in the on-chip cache. The third convolutional layer outputs 6 lines. The third / and subsequent data sets can be calculated according to the above logic to complete the calculation of the data to be inferred.

[0081] Therefore, the execution logic of this embodiment is simpler than that of depth-first search. In the above embodiment, only two stages are required, avoiding repeated logical checks and jumps.

[0082] Compared to breadth-first recalculation, the embodiments of this application can completely avoid redundant calculations.

[0083] Compared to breadth-first reuse, the target layer fusion strategy collection mechanism is simpler. By utilizing the sorting of the combinations of network layer outputs, it only needs to calculate the on-chip cache data usage when each fusion module outputs one line in each fusion method.

[0084] Optionally, information from some fusion modules can be recorded and reused in other fusion methods, further reducing computational load. This allows for feasibility assessment, avoiding exhaustive search or dynamic programming, whose algorithmic complexity is generally no less than K·n. 2 Where K represents the number of feasible h-dimensional or w-dimensional segmentation methods, n represents the number of network layers, and the algorithm complexity of this method is generally mn.

[0085] Based on the breadth-first and depth-first search schemes provided by the two related technologies mentioned above, this application offers a scheme that balances breadth and depth. Compared with the recomputation operation of breadth-first search, it avoids repeated computation of inference data; compared with depth-first search, it avoids repeated logical judgments and computations; and compared with the reuse operation of breadth-first search, the search mechanism is simpler, and the algorithm complexity is Kn.

[0086] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating a data processing method for an AI chip, provided by an exemplary embodiment of this application. This data processing method for an AI chip can be applied to the computer device shown above. Figure 5 In China, data processing methods used for AI chips include:

[0087] Step 510: Obtain the structural features of the neural network.

[0088] Among them, structural features are used to indicate the connection relationship between at least two network layers of the neural network, and the number of inputs and outputs of the network layers in the target dimension.

[0089] In this embodiment, the computer device can train and invoke the trained neural network. The computer device can load the neural network into an AI chip. After loading the neural network, the AI ​​chip can acquire the structural features of the neural network.

[0090] Structural features are used to indicate the connection relationships between at least two layers of a neural network. That is, structural features are used to indicate the topology of the neural network. Furthermore, structural features are also used to indicate the number of inputs and outputs of a network layer in the target dimension. In this application, the number of inputs and outputs is used to indicate the number of inputs and outputs of a network layer in the target dimension.

[0091] Optionally, the target dimension can be either a row dimension or a column dimension.

[0092] If a neural network consists of k layers, then the structural features can indicate the number of inputs and outputs for each layer in the target dimension. For example, if the target dimension is rows, then the structural features can indicate that the input layer of the neural network takes in 12 rows of data, the first convolutional layer has 12 rows of input and 10 rows of output, the second convolutional layer has 10 rows of input and 8 rows of output, and so on.

[0093] That is, the structural features in this application not only indicate the topology of the neural network, but also indicate the number of inputs and outputs of each network layer in the neural network.

[0094] Step 520: Based on structural features, at least two candidate inter-layer fusion strategies are determined. These candidate inter-layer fusion strategies are used to fuse network layers in the neural network.

[0095] In this embodiment, the AI ​​chip can determine at least two candidate inter-layer fusion strategies based on the aforementioned structural features. These candidate inter-layer fusion strategies are those that meet certain predetermined conditions. The inter-layer fusion strategies can fuse specified adjacent network layers in the neural network according to a specified number of blocks and a partitioning method.

[0096] For example, if the number of layer blocks is 1, the AI ​​chip will fuse all network layers into a single layer block. If the number of layer blocks is 2, the AI ​​chip will fuse all network layers into two layer blocks. If the neural network includes m network layers, then one layer block is formed by fusing k consecutive network layers, and the other layer block is formed by fusing (mk-2) consecutive network layers. It should be noted that the network layers described in this application do not include input and output layers.

[0097] In one possible approach, the AI ​​chip can identify at least two candidate inter-layer fusion strategies in ascending order of the target exchange volume.

[0098] It should be noted that the task of determining at least two candidate inter-layer fusion strategies can also be implemented by a CPU chip, and this application embodiment does not limit this.

[0099] Step 530: Determine the target inter-layer fusion strategy from at least two candidate inter-layer fusion strategies. The target exchange volume and peak inference memory consumption of the neural network after fusion using the target inter-layer fusion strategy meet the preset conditions. The target exchange volume is used to indicate the amount of data exchanged between the on-chip cache and off-chip storage of the AI ​​chip in the target dimension during a single inference process.

[0100] In one possible approach, the preset conditions include that the peak inference memory consumption is not less than the total on-chip cache capacity of the AI ​​chip, and that the target swap volume is the minimum value among the candidate inter-layer fusion strategies capable of inference on the AI ​​chip. Here, the peak inference memory consumption indicates the maximum value of the on-chip cache consumed by the neural network during inference.

[0101] In this embodiment, the AI ​​chip can determine the target inter-layer fusion strategy from at least two candidate inter-layer fusion strategies according to certain judgment logic. The first condition can be the minimum target exchange volume. It should be noted that the target exchange volume is a candidate inter-layer fusion strategy that has not yet been screened. For example, the AI ​​chip determines 10 candidate inter-layer fusion strategies in the first batch. Since three of these candidate inter-layer fusion strategies do not meet the criteria, they have been excluded. The AI ​​chip then selects the candidate inter-layer fusion strategy with the minimum target exchange volume from the remaining 7 candidate inter-layer fusion strategies.

[0102] After identifying the candidate inter-layer fusion strategy with the minimum target exchange volume, the AI ​​chip will further determine whether its hardware performance can meet the peak inference memory consumption required by the candidate inter-layer fusion strategy. If the AI ​​chip's hardware performance can meet the peak inference memory consumption required by the candidate inter-layer fusion strategy, it means that after the candidate inter-layer fusion strategy is integrated into the AI ​​chip with the neural network, it can normally process inference data and achieve the effect of the neural network performing inference work normally and efficiently.

[0103] It's important to note that the target exchange volume indicates the amount of data exchanged between the AI ​​chip's on-chip cache and off-chip storage in a single inference process. The target dimension can be one or two dimensions. If the target dimension is one dimension, it can be either a row or column dimension. If the target dimension has two dimensions, it includes both row and column dimensions. A single inference process indicates the process from when data is input from the input layer until it is output from the output layer.

[0104] For example, with Figure 3 The following example illustrates the concept of target exchange volume. Figure 4The neural network shown consists of an input layer, three consecutive convolutional layers, and an output layer. Based on this neural network, there are four fusion strategies between candidate layers, as shown in Table 1.

[0105]

[0106] Table 1

[0107] In this application, if the first candidate inter-layer fusion strategy is adopted, the fused neural network only needs to obtain inference data from the input layer, and then processes the inference data layer by layer using the entire layer block. During this process, data is no longer exchanged between the on-chip cache and off-chip storage of the AI ​​chip. The process of verifying the memory space required by this first candidate inter-layer fusion strategy is to assume that the last network layer of the entire neural network outputs one line of data, ensuring that the output layer has data output. In this case, determining the minimum memory space required by the first candidate inter-layer fusion strategy involves first determining the peak value of the memory space required for each layer block, and then determining the maximum value as the memory space required by the strategy. Since the third convolutional layer needs to output at least one line, it follows that the second convolutional layer needs to output at least three lines, the first convolutional layer needs to output at least five lines, and the input layer needs to input at least seven lines. During the entire neural network computation process, the most memory-intensive part is the computation process of the first convolutional layer. During this process, the memory space consumed includes seven lines of input data, five lines of output data, and two lines of boundary data required for each convolutional layer computation. Since there are three convolutional layers, the first candidate inter-layer fusion strategy requires at least 6 rows of boundary data. Based on the above analysis, it can be determined that the maximum memory consumption of the first candidate inter-layer fusion strategy is 18 rows.

[0108] In this application, if the second candidate inter-layer fusion strategy is adopted, the fused neural network receives inference data from the input layer. After the inference data is convolved by the first convolutional layer, it is not output but then input into the fused second convolutional layer. When the second convolutional layer obtains intermediate output data, the AI ​​chip transfers this intermediate data from the on-chip cache to off-chip storage. When this intermediate data is needed to be input into the third convolutional layer to obtain the corresponding result data, the AI ​​chip reads the intermediate data from off-chip storage into the on-chip cache, and then performs convolution calculations in the third convolutional layer before outputting the result data. Therefore, regardless of the candidate inter-layer fusion strategy used, the target exchange volume must always include the inference data input from the input layer and the result data obtained from the output layer.

[0109] The process of verifying the minimum memory space required by the second candidate inter-layer fusion strategy involves fusing the neural network according to the strategy, resulting in two layer blocks. The first block is formed by fusing the first and second convolutional layers, and the second block is formed by fusing the third convolutional layer. Then, the maximum memory consumption of each block is recorded. When the first block consumes the most memory, the second convolutional layer outputs one line, the first convolutional layer outputs three lines, and the input layer receives at least five lines. Therefore, the most memory-intensive time is during the computation of the first convolutional layer within the first block. This computation consumes 5 lines + 3 lines + 2 lines * 2, or 12 lines. Here, 5 lines represent the input of the first convolutional layer, 3 lines represent its output, and 2 lines represent the boundary data corresponding to the two convolutional layers retained in the first block. Each convolutional layer retains two lines. As described above, the first layer consumes a maximum of 12 lines of memory. Similarly, the computation of the second layer block consumes the most memory during the computation of the third convolutional layer, requiring 3 + 1 + 2 rows of memory, totaling 6 rows. In summary, the second candidate inter-layer fusion strategy consumes the most memory during the computation of the second layer block, with a maximum of 12 rows.

[0110] In this application, if the third candidate inter-layer fusion strategy is adopted, the calculation method is similar to that of the second candidate inter-layer fusion strategy. The target exchange quantity can be obtained as: input quantity (18 lines) + output quantity (12 lines) + exchange quantity (16 lines * 2), totaling 62 lines.

[0111] In this application, if the fourth candidate inter-layer fusion strategy is adopted, the calculation method is similar to that of the second candidate inter-layer fusion strategy. The target exchange quantity can be obtained as: input quantity (18 lines) + output quantity (12 lines) + exchange quantity (14 lines * 2 + 16 lines * 2), totaling 90 lines.

[0112] In this application, candidate inter-layer fusion strategies with smaller target exchange volumes can be given priority. That is, the AI ​​chip will prioritize detecting the feasibility of the first candidate inter-layer fusion strategy. Specifically, the AI ​​chip will prioritize detecting whether the first candidate inter-layer fusion strategy can fuse the neural network and complete the inference data computation in the AI ​​chip without memory overflow. In this example, among the four candidate inter-layer fusion strategies, the first candidate inter-layer fusion strategy has the smallest target exchange volume, followed by the second candidate inter-layer fusion strategy, then the third candidate inter-layer fusion strategy, and the fourth candidate inter-layer fusion strategy has the largest. Therefore, the AI ​​chip will sequentially detect whether each candidate inter-layer fusion strategy can serve as the target inter-layer fusion strategy, in the order of the first, second, third, and fourth candidate inter-layer fusion strategies.

[0113] When the AI ​​chip detects the peak inference memory consumption of the first candidate inter-layer fusion strategy, the computer device can either place the simulation process in the AI ​​chip or place the simulation process in the CPU chip for verification.

[0114] Specifically, if the AI ​​chip crashes due to memory overflow when detecting the peak inference memory consumption of the first candidate inter-layer fusion strategy, it indicates that the first candidate inter-layer fusion strategy is not the target candidate inter-layer fusion strategy. Optionally, since an AI chip crash may cause other uncontrollable harm to the system, this step can also be executed by the CPU, which stores the on-chip inference memory consumption peak threshold. For the first candidate inter-layer fusion strategy, after fusing the neural network using this strategy, a peak inference memory consumption can be obtained for each network layer computed. (The last sentence is incomplete and likely refers to a different implementation.) Figure 4 Taking the neural network model shown as an example, we will introduce the six computation times of the fused neural network. It should be noted that since the input data is input into the neural network twice, each network layer used for computation provides one computation time. Here, we take the example of the output layer outputting only one row for statistics. Please refer to Table 2 for details.

[0115]

[0116] Table 2

[0117] It should be noted that AI chips or CPU chips can determine the peak inference memory consumption from the above states. Clearly, among the states corresponding to the six time points, the data corresponding to the first time point represents the time point with the highest on-chip cache usage. That is, the on-chip cache usage corresponding to the first time point is the peak inference memory consumption of the first candidate inter-layer fusion strategy, which is 18 lines.

[0118] It should be noted that, in Table 2, row 7 represents the on-chip buffer occupied by the input data when the first convolutional layer computes the first input data; row 5 represents the on-chip buffer occupied by the output data when the first convolutional layer computes the first input data; and row 6 represents the edge data corresponding to each of the input layer, the first convolutional layer, and the second convolutional layer, with each layer having 2 rows of edge data. Therefore, the neural network occupies 2 * 3 = 6 rows of edge data. It should also be noted that the third item, number 6, at each time step in Table 2 represents the on-chip buffer occupied by the edge data in the neural network.

[0119] In the second row of data shown in Table 2, the first item represents the on-chip cache space occupied by the input data during network layer computation, the second item represents the on-chip cache space occupied by the output data during network layer computation, and the third item represents the on-chip cache space occupied by the edge data in the neural network.

[0120] Based on the above process, the computer device can determine the target inter-layer fusion strategy from at least two candidate inter-layer fusion strategies, thereby enabling the computer device to obtain the usable inter-layer fusion strategy with the minimum target exchange volume.

[0121] Regarding the peak inference memory consumption, in one scenario where the neural network does not include padding operations, the peak inference memory consumption is equal to the sum of the boundary occupancy, the number of inputs to the peak layer, and the number of outputs of the peak layer. The boundary occupancy is the sum of the boundary occupancy of each layer in the inter-layer fusion strategy, and the peak layer is the network layer in the inter-layer fusion strategy that has the largest number of inputs in the target dimension and the sum of the number of inputs.

[0122] In this example, the boundary occupancy can be the third value, 6, shown in Table 2, which is the sum of the boundary occupancy values ​​for each network layer. Taking Table 2 as an example, the peak layer is the first convolutional layer.

[0123] In another scenario, where the neural network includes padding operations, the peak inference memory consumption equals the sum of padding occupancy, boundary occupancy, the number of inputs to the peak layer, and the number of outputs to the peak layer.

[0124] It should be noted that when the network layer is a convolutional layer with a p-dimensional convolutional kernel and a stride of q, the boundary occupancy of the network layer is (pq), where p and q are positive integers.

[0125] In a practical application scenario, when the network layer is a max pooling layer, the convolution kernel corresponding to the max pooling layer is 2-dimensional, the stride corresponding to the max pooling layer is 2, and the boundary occupancy of the max pooling layer is 0.

[0126] In this example, padding occupancy is the sum of padding data occupancy for each layer in the neural network.

[0127] Step 540: Based on the target inter-layer fusion strategy, merge adjacent network layers in the neural network to obtain the fused neural network.

[0128] In one possible implementation, the computer device can instruct the AI ​​chip or CPU chip to fuse adjacent network layers in the neural network based on a target inter-layer fusion strategy.

[0129] Optionally, the computer device retains an unfused backup of the neural network. Alternatively, the computer device does not retain backups of the unfused network layers; instead, it directly fuses adjacent network layers of the original neural network to obtain the fused neural network. In the former scenario, the computer device can delete the fused neural network after the current inference is complete. In the latter scenario, the computer device saves the fused neural network after the current inference is complete.

[0130] Step 550: Process the data to be inferred based on the fused neural network.

[0131] In this application, the AI ​​chip in the computer device has already loaded the fused neural network. Therefore, the computer device can instruct the AI ​​chip, or cause the CPU chip and the AI ​​chip to cooperate, to input inference data into the fused neural network, thereby obtaining the inference result data.

[0132] In summary, the data processing method for AI chips provided in this embodiment can, after acquiring the structural features of the neural network, determine at least two candidate inter-layer fusion strategies based on these features, then select the target inter-layer fusion strategy that satisfies the minimum target exchange volume and whose peak inference memory consumption meets the total on-chip cache capacity of the AI ​​chip. Finally, adjacent network layers in the neural network are fused based on the target inter-layer fusion strategy to obtain a fused neural network, which is then used to process the data to be inferred. Since the target exchange volume is the amount of data exchanged between the on-chip cache and off-chip storage in a single inference operation along the target dimension, this embodiment can determine an available target inter-layer fusion strategy to fuse the neural network while minimizing the amount of on-chip and off-chip data exchange, thereby improving the inference efficiency of the neural network without changing the AI ​​chip hardware.

[0133] Based on the scheme disclosed in the previous embodiment, the terminal can also employ an algorithm that preserves boundary data to avoid redundant calculations during the inference process, and can initially select candidate inter-layer fusion strategies through a sorting method, thereby quickly obtaining the available inter-layer fusion strategy with the minimum target exchange volume. Please refer to the following embodiment.

[0134] Please see Figure 6 , Figure 6 This is a flowchart illustrating a data processing method for an AI chip, provided by another exemplary embodiment of this application. This data processing method for an AI chip can be applied to the terminal shown above. Figure 6 The data processing method used in this AI chip includes:

[0135] Step 601: Obtain the structural features of the neural network.

[0136] In this example, the execution process of step 601 is the same as that of step 510, and will not be described again here.

[0137] Step 602: Based on structural features, m candidate inter-layer fusion strategies are obtained in ascending order of the number of blocks in the fused network layers.

[0138] The number of network layers is a positive integer, and m is an integer greater than or equal to 2.

[0139] It should be noted that, in this application, the number of layers in a neural network indicates the number of layers formed after the network layers used for computation are fused together. For example, if all the network layers used for computation in neural network A are fused into one layer, then the number of layers in this neural network is one. If all the network layers used for computation in neural network B are fused into two layers, then the number of layers in this neural network is two.

[0140] In this embodiment, for a given neural network, the number of available candidate inter-layer fusion strategies increases with the increase in the number of network layers used for computation. If the neural network includes one input layer, L network layers used for computation, and one output layer, then there are a total of 2 candidate inter-layer fusion strategies that can be used for inter-layer fusion of this neural network. L-1 There are 1, where L is an integer greater than or equal to 2.

[0141] In one possible approach, the AI ​​chip could handle all 2 L-1 The candidate inter-layer fusion strategies are ordered in ascending order of the number of blocks in the fused network layer, resulting in m candidate inter-layer fusion strategies. It should be noted that m ≤ 2. L-1 .

[0142] In another possible approach, AI chips are able to [achieve a certain level of capability]. L- Among the candidate inter-layer fusion strategies, the top m candidate inter-layer fusion strategies are selected in ascending order of the number of blocks in the fused network layers.

[0143] In one possible approach, step 602 can be performed in place of steps (a1) and (a2).

[0144] Step (a1) is to obtain n in response to the number of network layers consisting of i blocks. i There are n candidate inter-layer fusion strategies, where m equals n. i And n i It is a positive integer.

[0145] Step (a2): Arrange n according to the increasing order of the target exchange quantity. iWe obtain m candidate inter-layer fusion strategies and get the i-th queue, which is used to provide m candidate inter-layer fusion strategies.

[0146] In this example, after the AI ​​chip completes the above steps (a1) and (a2), the AI ​​chip is able to arrange candidate fusion strategies with the same number of layers, and the candidate fusion strategies are arranged in order of increasing target exchange volume.

[0147] In one possible implementation, assuming the neural network has L layers, then there are a total of 2 possible inter-layer fusion methods. L-1 The algorithm calculates and sorts the feature maps of each species. This sorting process is time-consuming when L is large. However, the algorithm complexity of this method is K*L.

[0148] In another possible approach, the AI ​​chip may continue to execute steps (a3), (a4), and (a5) in place of step 602.

[0149] Step (a3) ​​responds to the network layer consisting of (i+1) blocks, resulting in n. i+1 There are n candidate inter-layer fusion strategies, where m is greater than or equal to n. i .

[0150] In this example, when the number of blocks in the network increases from i to i+1, the AI ​​chip can obtain a new candidate inter-layer fusion strategy corresponding to the number of blocks i+1.

[0151] Step (a4): Based on the increasing order of the target exchange quantities, n... i+1 The candidate inter-layer fusion strategies are arranged in the i-th queue to obtain the (i+1)-th queue.

[0152] In this example, since the number of candidate inter-layer fusion strategies obtained at this time is limited, the AI ​​chip can directly arrange the candidate inter-layer fusion strategies according to the number of times the target exchange volume increases, and obtain the (i+1)th queue.

[0153] Step (a5), in response to The candidate inter-layer fusion strategies in the (i+1)th queue are taken as the m1 candidate inter-layer fusion strategies.

[0154] In this example, the number of candidate inter-layer fusion strategies is the sum of the candidate inter-layer fusion strategies corresponding to each type of layer block. In this scenario, the AI ​​chip uses the candidate inter-layer fusion strategies in the (i+1)th queue as the m1th candidate inter-layer fusion strategy, thereby achieving rapid initial selection of candidate inter-layer fusion strategies and improving the efficiency of initializing the fused neural network.

[0155] In one possible implementation, in permutation ni+1 When considering multiple candidate inter-layer fusion strategies, the AI ​​chip can choose a first threshold as the standard, and further refine the selection from n... i+1 Select the desired inter-layer fusion strategy from among the candidate inter-layer fusion strategies.

[0156] Alternatively, in one possible approach, the first threshold is a constant. In another possible approach, the first threshold is the maximum value of the target exchange amount of the candidate inter-layer fusion strategies in the first queue.

[0157] As an alternative implementation, step 602 can be replaced by the following steps (b1) and (b2) to determine the target inter-layer fusion strategy.

[0158] Step (b1) is to determine the inter-layer fusion strategy with the least target exchange volume among the m candidate inter-layer fusion strategies as the intermediate inter-layer fusion strategy.

[0159] In this example, the AI ​​chip first needs to determine the candidate inter-layer fusion strategy with the minimum target exchange from among the candidate inter-layer fusion strategies. Since reducing the target exchange can significantly improve the efficiency of neural network inference, improving the inference capability of the neural network is a priority requirement of this application.

[0160] In step (b2), in response to the peak inference memory consumption of the intermediate layer fusion strategy being less than or equal to the total on-chip cache capacity of the AI ​​chip, the intermediate layer fusion strategy is determined as the target inter-layer fusion strategy.

[0161] In this example, based on the intermediate layer fusion strategy that minimizes the target exchange volume, the AI ​​chip can continue to detect the peak inference memory consumption of the intermediate layer fusion strategy. The process for detecting the peak inference memory consumption can be found in Table 2 of step 530, and will not be repeated here.

[0162] As an alternative implementation, step 602 can be replaced by the following steps (c1) and (c2) to determine the target inter-layer fusion strategy.

[0163] Step (c1) is to determine the inter-layer fusion strategy with the minimum target exchange volume among the m candidate inter-layer fusion strategies as the intermediate inter-layer fusion strategy.

[0164] In step (c2), in response to the peak inference memory consumption of the intermediate inter-layer fusion strategy being greater than the total on-chip cache capacity of the AI ​​chip, the intermediate inter-layer fusion strategy is removed from at least two candidate inter-layer fusion strategies.

[0165] In this example, steps (c1) and (c2) are used to remove those that have been judged and cannot be used as the target inter-layer fusion strategy from the queue, in order to save storage space and facilitate the rapid determination of the target inter-layer fusion strategy.

[0166] In this embodiment of the application, after the AI ​​chip completes step 602, it can execute step 603, as well as steps 604 and 605.

[0167] Step 603: Determine the target inter-layer fusion strategy from at least two candidate inter-layer fusion strategies.

[0168] In this example, the execution process of step 603 can be found in step 530, and will not be repeated here.

[0169] Step 604: If the peak inference memory consumption of the neural network after fusion according to the candidate inter-layer fusion strategy corresponding to the number of layers is less than the total on-chip cache capacity of the AI ​​chip, then the data combination layer is determined from the network layers of the neural network.

[0170] The data combination layer processes the input data to obtain the output data when the corresponding input data equals the upper limit of the input; the sequence number of the data combination layer is greater than or equal to the sequence number of the network layer corresponding to the downsampling operation.

[0171] Step 605: In response to the inter-layer fusion strategy of the data combination layer satisfying the total on-chip cache capacity of the AI ​​chip, the inter-layer fusion strategy of the data combination layer is determined as the target inter-layer fusion strategy.

[0172] In this embodiment, the AI ​​chip can change the data processing method during neural network fusion and add a data combination layer to make the fused neural network form a tree structure. This enables the candidate layer fusion strategy to meet the on-chip inference memory consumption peak threshold when the peak inference memory consumption is equal to or greater than the total on-chip cache capacity of the AI ​​chip, thereby improving the inference efficiency of the fused neural network.

[0173] Step 606: Based on the target inter-layer fusion strategy, merge adjacent network layers in the neural network to obtain the fused neural network.

[0174] In this example, the execution method of step 606 can be found in the execution method of step 540. For details, please refer to the execution method of step 540. It will not be repeated here.

[0175] Step 607: Process the data to be inferred based on the fused neural network.

[0176] In this example, the execution method of step 607 can be found in the execution method of step 550. For details, please refer to the execution method of step 550. It will not be repeated here.

[0177] In summary, this embodiment can select a target inter-layer fusion strategy that meets the peak inference memory consumption of the AI ​​chip's on-chip cache capacity while minimizing the target exchange volume. By using the target inter-layer fusion strategy, the neural network is fused, enabling the fused neural network to have high inference efficiency without changing the hardware.

[0178] The data processing method for AI chips provided in this embodiment can also quickly filter out a subset of candidate inter-layer fusion strategies with smaller target exchange volumes from multiple candidate inter-layer fusion strategies by setting a threshold, thereby improving the speed of determining the target inter-layer fusion strategy and thus improving the speed of initializing the neural network in the AI ​​chip.

[0179] The data processing method for AI chips provided in this embodiment can also input the batch-obtained input data after combining it in the data combination layer when the inference memory consumption peak of the corresponding candidate inter-layer fusion strategy does not meet the total on-chip cache capacity of the AI ​​chip, so that the inference memory consumption peak of the candidate inter-layer fusion strategy meets the total on-chip cache capacity of the AI ​​chip. This improves the efficiency of neural networks including downsampling operations in inter-layer fusion and increases the inference speed after inter-layer fusion is completed.

[0180] In one possible implementation provided in this application, the AI ​​chip can execute the core idea of ​​this application in two steps. First, the AI ​​chip sorts all possible fusion methods according to the data exchange volume between on-chip cache and off-chip storage based on the input size of the network layer. Second, the AI ​​chip sequentially checks the feasibility of each fusion method in the sorted order until a feasible fusion method is found that minimizes the data exchange volume between the AI ​​chip's on-chip cache and off-chip storage.

[0181] Please see Figure 7 , Figure 7 This is a schematic diagram of the hardware execution logic of a fused neural network with padding operations, as provided in this application. In another possible implementation, this application describes the execution logic for a neural network with padding operations. This execution logic includes three stages: stage one, stage two, and stage three. Assume the number of layers is one. Wherein, Figure 7 The leftmost and rightmost rows of each block layer are fill layers, not inputs to the network layers.

[0182] In Phase 1, the first set of data does not have pre-reserved boundary data in the on-chip cache. However, a padding operation occurs within the neural network, the specific process of which is as follows:

[0183] The first input data consists of 9 lines (the 3rd + 4th + 2nd lines from left to right of the bottom edge of the first convolutional layer). The AI ​​chip uses the corresponding 1 line of padding data (the leftmost line of the first convolutional layer) and stores the corresponding 2 lines of boundary data (the part corresponding to the bold triangle on the left side of the bottom edge of the first convolutional layer) in the on-chip cache. The first convolutional layer outputs 8 lines (the 4th + 2nd + 2nd lines from left to right of the top edge of the first convolutional layer). The first convolutional layer uses 1 line of padding data and stores the 2 lines of boundary data (the triangle on the bold left side of the top edge of the first convolutional layer) in the on-chip cache. The second convolutional layer outputs 7 lines (the 5th + 2nd lines from left to right of the top edge of the second convolutional layer). It uses 1 line of padding data and stores the 2 lines of boundary data (the part corresponding to the bold triangle on the left side of the top edge of the second convolutional layer) in the on-chip cache. The third convolutional layer outputs 6 lines.

[0184] Phase Two, the second / third data set / and other intermediate data, requires the use of pre-reserved boundary data. The specific process is as follows:

[0185] The second set of input data consists of 6 lines. The 2 lines of boundary data corresponding to the reserved input layer are reused, and the last 2 lines of this second set of input data are stored in the on-chip cache. The first convolutional layer outputs 6 lines, reusing the 2 lines of boundary data corresponding to the reserved first convolutional layer, and storing the last 2 lines of this first convolutional layer as boundary data in the on-chip cache. The second convolutional layer outputs 6 lines, reusing the 2 lines of boundary data corresponding to the reserved second convolutional layer, and storing the last 2 lines of this second convolutional layer as boundary data in the on-chip cache. The third convolutional layer outputs 6 lines. The third set of data / and other intermediate data can be calculated according to this logic.

[0186] Phase three, the final data set, requires the use of pre-saved boundary data and filler data. The specific process is as follows:

[0187] The second input data consists of 3 rows. One row is used for padding, and the AI ​​chip reuses the 2 rows of boundary data corresponding to the retained input layer. The first convolutional layer outputs 4 rows. The AI ​​chip uses one row for padding and reuses the 2 rows of boundary data corresponding to the retained first convolutional layer. The second convolutional layer outputs 5 rows. The AI ​​chip uses one row for padding and reuses the 2 rows of boundary data corresponding to the retained second convolutional layer. The third convolutional layer outputs 6 rows.

[0188] Generally speaking, the execution logic of this method does not exceed three stages, namely the three cases of the first data / intermediate data / last data. In special cases, such as when there are irregular shape ops like resize bilinear, when the common factor of the input shape and the output shape is large, there may be no more than the number of copies of the common factor in the intermediate data, but there is no judgment and jump logic inside each fusion module.

[0189] In another possible execution mode provided by this application, the AI chip can quickly obtain candidate inter-layer fusion strategies through the following practical methods.

[0190] Suppose there are n network layers for computing in the neural network, then there are 2 n-1 types of all possible candidate inter-layer fusion strategies. The AI chip calculates their feature maps and sorts them. When n is relatively large, the process of determining the candidate inter-layer fusion strategy will consume a lot of time. But the algorithm complexity of this method is m·n.

[0191] Suppose the network has a total of 11 tensors: input / layer1_output / layer2_output / … / layer10_output. Excluding the input layer that must be loaded and layer10_output that must be stored, assume layer9_output < layer8_output < … < layer2_output < layer1_output. It should be noted that the output in the above tensors represents the size of the output quantity. The steps of this method are as follows:

[0192] Step 1, select the first-round candidate inter-layer fusion strategy, such as the candidate inter-layer fusion strategy of only swapping in and out one network layer, add it to the list, and sort it, layer9_output < layer8_output < … < layer2_output < layer1_output.

[0193] Step 2: Determine whether there is a method smaller than layer1_output among the candidate inter-layer fusion strategies for swapping in and out 2 network layers (layers), and add it to the list. Then re-sort. Assume that the total sorting of the candidate inter-layer fusion strategies after addition is layer9_output < layer8_output <... layer2_output < layer9_output + layer8_output < layer9_output + layer7_output < layer8_output + layer7_output < layer1_output. The actual addition process is as follows:

[0194] Step 2.1: When the AI chip actually executes, first judge the combination of layer9_output and other network layers until no smaller combination of layer9_output and other network layers can be found. For example, if the AI chip finds that layer9_output + layer6_output > layer1_output, it stops adding the combination of layer9_output and other network layers to the list.

[0195] Step 2.2: Similarly, find the combination of layer8_output and other network layers, the combination of layer7_output and other network layers,..., until no combination of two network layers can be found. For example, if the combination of layer7_output and other layers > layer1_output, it stops adding the combination of two layers to the list.

[0196] Step 3: Similarly, determine whether there is a method smaller than layer1_output among the candidate inter-layer fusion strategies for swapping in and out 3 network layers. For example, assume that only layer7_output + layer8_output + layer9_output < layer1_output exists. The total sorting after addition is layer9_output < layer8_output <... layer2_output < layer7_output + layer8_output < layer7_output + layer9_output < layer8_output + layer9_output < layer7_output + layer8_output + layer9_output < layer1_output.

[0197] Step 4: Assume that in the combination of 4 network layers, no method smaller than layer1_output can be found, then stop adding the combination of multiple layers to the list.

[0198] Step 5: Determine the feasibility of the candidate inter-layer fusion strategies in the list one by one. If a feasible method exists, the process provided in this embodiment is terminated.

[0199] Step 6: Determine the feasibility of each candidate inter-layer fusion strategy in the list. If no feasible method exists, select the next candidate inter-layer fusion strategy that has not yet been evaluated. For example, select "all remaining candidate inter-layer fusion strategies for switching in and out two network layers", and determine whether there is a fusion method with 3 layers / 4 layers / ... less than layer1_output + layer2_output.

[0200] Please refer to Figure 8 , Figure 8 This is a schematic diagram of a tree-structured processing logic provided in an embodiment of this application. Figure 8 In this structure, the first network layer 810, the nth network layer 820, and the mth network layer 830 form a tree-like processing structure, which is used to support neural networks including downsampling operations.

[0201] The size of the data to be inferred is n*h*w*c. Assume the total space size is also n*h*w*c, where n represents the number of inputs, h represents the number of rows, w represents the number of columns, and c represents the number of channels. After k downsampling iterations, the data size will become 2^k. -k n*h*w*c.

[0202] If the data to be inferred after k downsampling cycles is still stored in the on-chip cache, then the remaining space in the on-chip cache is (1-2 -k )n*h*w*c.

[0203] exist Figure 8 In the tree structure shown, after multiple downsampling operations, the data processed by the nth network layer 820 (from the first network layer 810 to the nth network layer 820) becomes smaller than the data processed by the first network layer 810. The AI ​​chip stores the first data from the nth network layer 820 in the on-chip cache, and uses the remaining space in the on-chip cache to continue calculating the second data from the first network layer 810 to the nth network layer 820, storing it in the on-chip cache as well, and so on.

[0204] Assuming the neural network has accumulated k data points from the nth layer (820), the AI ​​chip then uses these k data points from the nth layer (820) to calculate layer by layer up to the mth layer (830), thus forming... Figure 8 The tree-like data processing logic structure is shown.

[0205] Please refer to Figure 9 , Figure 9 This is a schematic diagram of a tree-like data processing flow provided in an embodiment of this application. In this embodiment, the AI ​​chip can use the following method to confirm a feasible tree-like data processing structure.

[0206] Step 1: Add all combinations of downsample_output, that is, all possible "tree-like data processing fusion strategies", to a list according to the complexity of the combination (a single downsample_output represents a first-level tree, and a combination of multiple downsample_outputs represents a multi-level tree).

[0207] Step two: Determine the feasibility of each tree-based data processing and fusion method in turn until a method is found that can normally infer the data to be inferred in the neural network. This method can avoid the amount of data exchange between on-chip cache and off-chip storage.

[0208] When AI chips determine the feasibility of tree-based processing fusion methods, each fusion module is also required to output at least one line of data. For example, in Figure 9 In the example shown, the input layer must output at least one line of data up to the second max pooling layer (max6), and the second max pooling layer (max6) must output at least one line of data up to the fourth max pooling layer (max12).

[0209] exist Figure 9 The network consists of 8 convolutional layers (conv1 to conv8) and 4 max-pooling layers (max1 to max4). Assume the total on-chip cache capacity is: 28 rows of input + 26 rows of conv1 + 16 rows of max2 + boundary conditions (2 rows of input + 2 rows of conv1 + 2 rows of max1 + 2 rows of max3). If all network layers are merged, max4 would need at least 1 output row. Conversely, input would need 76 rows, and conv1 would need 74 output rows. Clearly, the total on-chip cache capacity is insufficient, preventing the merging of all network layers and inevitably leading to data exchange between the on-chip cache and off-chip storage.

[0210] When using this method, assume the data is divided into four parts by rows. The first part of the data is input from the input layer (input), with 28 rows of boundary data stored in the on-chip buffer. The first convolutional layer (conv1) outputs 26 rows, with 2 rows of boundary data stored in the on-chip buffer. The second convolutional layer (conv2) outputs 24 rows, and no data needs to be stored in the second convolutional layer (conv2). The first max pooling layer (max1) outputs 12 rows, with 2 rows of boundary data stored in the on-chip buffer. The third convolutional layer (conv3) outputs 10 rows, with 2 rows of boundary data stored in the on-chip buffer. The fourth convolutional layer (conv4) outputs 8 rows, and no data needs to be stored in the fourth convolutional layer (conv4). The second max pooling layer (max2) outputs 4 rows, all of which are stored in the on-chip buffer. The remaining space is then used to process the other parts of the data.

[0211] The second / third / fourth data sets are input into the input layer (16 lines). Two lines of boundary data are reused and retained. The last two lines of data from this input layer are stored in the on-chip buffer. The first convolutional layer (conv1) outputs 16 lines. The two lines of boundary data from conv1 are reused and stored in the on-chip buffer. The second convolutional layer (conv2) outputs 16 lines. No data needs to be retained. The first max-pooling layer (max1) outputs 8 lines. The reserved data from the first max-pooling layer (max1) is reused. The first max pooling layer (max1) outputs 2 rows of boundary data, and the last 2 rows of boundary data are kept in the on-chip cache. The third max pooling layer (conv3) outputs 8 rows, reuses the 2 rows of boundary data from the third max pooling layer (conv3), and keeps the 2 rows of boundary data from the third max pooling layer (conv3) in the on-chip cache. The fourth convolutional layer (conv4) outputs 8 rows, and no data needs to be kept in the fourth convolutional layer (conv4). The second max pooling layer (max2) outputs 4 rows, all of which are kept in the on-chip cache. The remaining space is used to process other data.

[0212] Because the total on-chip cache space is 28 rows of input layer + 26 rows of the first convolutional layer (conv1) + 16 rows of the second max pooling layer (max2) + boundary preservation (2 rows of input layer + 2 rows of the first convolutional layer (conv1) + 2 rows of the first max pooling layer (max1) + 2 rows of the third max pooling layer (conv3), the remaining space can be used to calculate the second / third / fourth sets of data. At this point, the second max pooling layer (max2) has collected a total of 16 rows of data, which can be used for subsequent calculations. The fifth convolutional layer (conv5) outputs 14 rows, the sixth convolutional layer (conv6) outputs 12 rows, the third max pooling layer (max3) outputs 6 rows, the seventh convolutional layer (conv7) outputs 4 rows, the eighth convolutional layer (conv8) outputs 2 rows, and the fourth max pooling layer (max4) outputs 1 row.

[0213] This application can also use multi-level trees. For example, the input layer to the second max pooling layer (max6) can be a fusion module, the second max pooling layer (max6) to the third max pooling layer (max9) can be a fusion module, and the third max pooling layer (max9) to the fourth max pooling layer (max12) can be a fusion module.

[0214] Continuing with the example above, after processing the first and second sets of data, the AI ​​chip does not calculate the third set of data. Instead, it inputs the first two sets of data into the fifth convolutional layer (conv7). At this point, the second max pooling layer (max6) has collected a total of 8 rows of data. The second max pooling layer (max6) retains the last 2 rows of data. The fifth convolutional layer (conv7) outputs 6 rows of data, retaining the last 2 rows of data from the fifth convolutional layer (conv7). The sixth convolutional layer (conv8) outputs 4 rows of data, retaining no data from the sixth convolutional layer (conv8). The third max pooling layer (max9) outputs 2 rows of data, all of which are stored in the on-chip cache.

[0215] The AI ​​chip then uses the third / fourth data set to input the third / fourth data set into the fifth convolutional layer (conv7). At this point, the sixth convolutional layer (conv8) collects another 8 rows of data, reuses the last 2 rows of data retained by the second max pooling layer (max6), the fifth convolutional layer (conv7) outputs 8 rows of data, reuses the last 2 rows of data retained by the fifth convolutional layer (conv7), the sixth convolutional layer (conv8) outputs 8 rows of data, and does not retain any data. The third max pooling layer (max9) outputs 4 rows of data, all of which are stored in the on-chip cache.

[0216] The third max pooling layer (max9) outputs a total of 6 rows of data. The AI ​​chip continues to calculate from the third max pooling layer (max9) to the fourth max pooling layer (max12). The seventh convolutional layer (conv10) outputs 4 rows of data, the eighth convolutional layer (conv11) outputs 2 rows of data, and the fourth max pooling layer (max12) outputs 1 row of data.

[0217] In this embodiment, multi-level trees can function correctly within a smaller on-chip cache space. For example, if the total on-chip cache space is 28 rows of data from the input layer + 26 rows of data from the first convolutional layer + 8 rows of data from the second max pooling layer (max6) + 6 rows of data from the third max pooling layer (max9) + the total reserved boundary data (2 rows of boundary data corresponding to the input layer + 2 rows of boundary data corresponding to the first convolutional layer + 2 rows of boundary data corresponding to the first max pooling layer + 2 rows of boundary data corresponding to the third convolutional layer + 2 rows of boundary data corresponding to the second max pooling layer + 2 rows of boundary data corresponding to the fifth convolutional layer), then a two-level tree is feasible (the space occupied by one row of the third max pooling layer is less than half the space occupied by one row of the second max pooling layer).

[0218] In practical applications, on the one hand, neural networks for classification purposes typically employ a convolution + downsampling pattern repeatedly. On the other hand, neural networks for denoising purposes also repeatedly employ a convolution + downsampling pattern in the first half of the neural network (e.g., the downsampling in the first half of a fully convolutional network like UET).

[0219] Furthermore, for complex scenarios, AI chips can combine tree-like data processing structures and candidate inter-layer fusion strategies to determine whether the peak inference memory consumption exceeds the total on-chip cache capacity of the AI ​​chip. For example, when determining the feasibility of a candidate inter-layer fusion strategy, if the peak inference memory consumption of the candidate inter-layer fusion strategy does not meet the total on-chip cache capacity of the AI ​​chip, but there are one or more downsampling operations in the neural network, the method described in the tree-like data processing structure can be used to split the fusion module in the original neural network into multiple fusion modules of the tree-like data processing structure. Further, it can be determined whether the peak inference memory consumption of the neural network after modifying the data processing structure meets the total on-chip cache capacity of the AI ​​chip. If feasible, that is, the candidate inter-layer fusion strategy after using the tree-like data processing structure can be determined as the target inter-layer fusion strategy.

[0220] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0221] Please refer to Figure 10 , Figure 10 This is a structural block diagram of a data processing apparatus for an AI chip, provided in an exemplary embodiment of this application. This data processing apparatus for the AI ​​chip can be implemented as all or part of a terminal through software, hardware, or a combination of both. The apparatus includes:

[0222] The feature acquisition module 1010 is used to acquire the structural features of the neural network.

[0223] The first determining module 1020 is used to determine at least two candidate inter-layer fusion strategies based on the structural features, wherein the candidate inter-layer fusion strategies are used to fuse the network layers in the neural network.

[0224] The second determining module 1030 is used to determine a target inter-layer fusion strategy from at least two candidate inter-layer fusion strategies. The target exchange volume and the peak inference memory consumption of the neural network after fusion using the target inter-layer fusion strategy meet preset conditions. The target exchange volume is used to indicate the amount of data exchanged between the on-chip cache and off-chip storage of the AI ​​chip in the target dimension during a single inference process. The peak inference memory consumption is used to indicate the maximum value of the on-chip cache consumed by the neural network during the inference process.

[0225] The network fusion module 1040 is used to fuse adjacent network layers in the neural network based on the target inter-layer fusion strategy to obtain a fused neural network.

[0226] The data processing module 1050 is used to process the data to be inferred based on the fused neural network.

[0227] In an optional embodiment, the preset conditions involved in the device include that the peak inference memory consumption is not less than the total on-chip cache capacity of the AI ​​chip, and that the target exchange volume is the minimum value among the candidate inter-layer fusion strategies that can be inferred on the AI ​​chip.

[0228] In an optional embodiment, the first determining module 1020 is used to obtain m candidate inter-layer fusion strategies based on the structural features and in ascending order of the number of blocks in the fused network layers, where the number of blocks in the network layers is a positive integer and m is an integer greater than or equal to 2.

[0229] In an optional embodiment, the first determining module 1020 is configured to determine when m1 equals n. i In the case where the number of blocks in the network layer is i, n is obtained. i The candidate inter-layer fusion strategies, ni The integer is a positive integer; based on the increasing order of the target exchange quantity, n is arranged... i The candidate inter-layer fusion strategies are used to obtain the i-th queue, which is used to provide m1 candidate inter-layer fusion strategies; n i It is the number of candidate inter-layer fusion strategies for a layer block number of i.

[0230] In an optional embodiment, the apparatus further includes a third determining module for determining when m is greater than n. i In the case where the number of blocks in the network layer is (i+1), n ​​is obtained. i+1 The candidate inter-layer fusion strategies are defined; based on the increasing order of the target exchange volume, n... i+1 The candidate inter-layer fusion strategies are arranged in the i-th queue to obtain the (i+1)-th queue; in response to The candidate inter-layer fusion strategies in the (i+1)th queue are taken as m candidate inter-layer fusion strategies.

[0231] In an optional embodiment, the n involved in the device i+1 The target exchange amount of each candidate inter-layer fusion strategy is less than a first threshold, which is a constant or the maximum value of the target exchange amount of the candidate inter-layer fusion strategies in the first queue.

[0232] In an optional embodiment, the second determining module 1030 is configured to determine the candidate inter-layer fusion strategy with the least target exchange volume among the m candidate inter-layer fusion strategies as an intermediate inter-layer fusion strategy; and to determine the intermediate inter-layer fusion strategy as the target inter-layer fusion strategy in response to the fact that the peak inference memory consumption of the intermediate inter-layer fusion strategy is less than or equal to the total on-chip cache capacity of the AI ​​chip.

[0233] In an optional embodiment, the second determining module 1030 is configured to determine the candidate inter-layer fusion strategy with the least target exchange volume among the m candidate inter-layer fusion strategies as an intermediate inter-layer fusion strategy; in response to the inference memory consumption peak of the intermediate inter-layer fusion strategy being greater than the total on-chip cache capacity of the AI ​​chip, the intermediate inter-layer fusion strategy is deleted from at least two of the candidate inter-layer fusion strategies.

[0234] In an optional embodiment, when the neural network does not include padding operations, the peak inference memory consumption is equal to the sum of boundary occupancy, the number of inputs to the peak layer, and the number of outputs of the peak layer. The boundary occupancy is the sum of the boundary occupancy of each layer in the inter-layer fusion strategy, and the peak layer is the network layer in the inter-layer fusion strategy that has the largest number of inputs and the sum of the number of inputs in the target dimension.

[0235] In an optional embodiment, when the neural network includes a padding operation, the peak inference memory consumption is equal to the sum of the padding occupancy, the boundary occupancy, the number of inputs to the peak layer, and the number of outputs of the peak layer.

[0236] In an optional embodiment, when the network layer is a convolutional layer and the corresponding convolutional kernel is p-dimensional with a stride of q, the boundary occupancy of the network layer is (pq).

[0237] In an optional embodiment, the apparatus further includes an execution module, configured to, when the peak inference memory consumption is equal to or greater than the total on-chip cache capacity of the AI ​​chip, and the neural network includes a downsampling operation, determine a data combination layer from the network layers of the neural network if the peak inference memory consumption of the neural network after fusion according to the candidate inter-layer fusion strategy corresponding to the number of layers of 1 is less than the total on-chip cache capacity of the AI ​​chip; wherein the data combination layer processes the input data to obtain output data when the obtained input data is equal to the upper limit of the input; wherein the sequence number of the data combination layer is greater than or equal to the sequence number of the network layer corresponding to the downsampling operation; and in response to the inter-layer fusion strategy of the data combination layer satisfying the total on-chip cache capacity of the AI ​​chip, determine the inter-layer fusion strategy of the data combination layer as the target inter-layer fusion strategy.

[0238] In an optional embodiment, the structural features of the device are used to indicate the connection relationship between at least two network layers of the neural network, and the number of inputs and outputs of the network layers in the target dimension.

[0239] In summary, this embodiment can select a target inter-layer fusion strategy that meets the peak inference memory consumption of the AI ​​chip's on-chip cache capacity while minimizing the target exchange volume. By using the target inter-layer fusion strategy, the neural network is fused, enabling the fused neural network to have high inference efficiency without changing the hardware.

[0240] The data processing device for AI chips provided in this embodiment can also quickly filter out a subset of candidate inter-layer fusion strategies with smaller target exchange volume from multiple candidate inter-layer fusion strategies by setting a threshold threshold, thereby improving the speed of determining the target inter-layer fusion strategy and thus improving the speed of initializing the neural network of the AI ​​chip.

[0241] The data processing apparatus for AI chips provided in this embodiment can also input the batch-obtained input data after combining it in the data combination layer when the inference memory consumption peak of the corresponding candidate inter-layer fusion strategy does not meet the total on-chip cache capacity of the AI ​​chip, so that the inference memory consumption peak of the candidate inter-layer fusion strategy meets the total on-chip cache capacity of the AI ​​chip. This improves the efficiency of neural networks including downsampling operations in inter-layer fusion and increases the inference speed after inter-layer fusion is completed.

[0242] This application also provides a computer-readable medium storing at least one instruction, which is loaded and executed by the processor to implement the data processing method for AI chips as described in the above embodiments.

[0243] It should be noted that the data processing device for AI chips provided in the above embodiments is only illustrated by the division of the above functional modules when executing the data processing method for AI chips. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device for AI chips provided in the above embodiments and the data processing method embodiments for AI chips belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0244] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0245] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0246] The above description is merely an exemplary embodiment that can be implemented in this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method for an artificial intelligence (AI) chip, comprising: The method comprises: obtaining a structural feature of a neural network; based on the structural feature, determining at least two candidate inter-layer fusion strategies for fusing network layers in the neural network; from the at least two candidate inter-layer fusion strategies, determining a target inter-layer fusion strategy, wherein a target exchange amount and a peak inference memory consumption of the neural network after fusion using the target inter-layer fusion strategy meet a preset condition in an inference process, the target exchange amount is used to indicate a data exchange amount between an on-chip cache and off-chip storage of the AI chip in a target dimension in a single inference process, the peak inference memory consumption is used to indicate a maximum value of the on-chip cache consumed by the neural network in the inference process, and the preset condition comprises that the peak inference memory consumption is less than a total capacity of the on-chip cache of the AI chip, and the target exchange amount is a minimum value in the candidate inter-layer fusion strategies for inference on the AI chip; based on the target inter-layer fusion strategy, fusing adjacent network layers in the neural network to obtain a fused neural network; based on the fused neural network, processing to-be-inferred data.

2. The method of claim 1, wherein, The method comprises: Based on the structural features, in ascending order of the number of layer blocks composed of the network layers after fusion, obtain The candidate inter-layer fusion strategies are obtained, and the number of layer blocks composed of the network layers is a positive integer, is an integer greater than or equal to 2.

3. The method of claim 2, wherein, is the number of the candidate inter-layer fusion strategies with the layer block number i, and is equal to In the case that obtaining, based on the structure feature, the candidate inter-layer fusion strategies in ascending order of the layer block number of the fused network layer, comprises: In response to the number of layer blocks of the network layer composition being , obtaining candidate inter-layer fusion strategies, is a positive integer; Arranged in ascending order of the target exchange volume The candidate inter-layer fusion strategy is obtained to obtain the first... Queue, the first Queues are used to provide The proposed inter-layer fusion strategies.

4. The method of claim 3, wherein, In greater than the case where the first queue is obtained, the method further comprises: In response to the number of layer blocks of the network layer composition being , obtaining candidate inter-layer fusion strategies. Based on the ascending order of the target exchange volume, The candidate inter-layer fusion strategies are arranged into the first... In the queue, the first one is obtained. queue; in response to the first the candidate inter-layer fusion strategy in the queue as the candidate inter-layer fusion strategy.

5. The method of claim 4, wherein, the target switching volume of the candidate inter-layer fusion strategy is less than a first threshold value, and the first threshold value is a constant or a maximum value of the target switching volume of the candidate inter-layer fusion strategy in the first queue.

6. The method of claim 2, wherein, The method comprises: Will The candidate inter-layer fusion strategy with the lowest target exchange volume among the candidate inter-layer fusion strategies is determined as the intermediate inter-layer fusion strategy. in response to the peak inference memory consumption of the intermediate inter-layer fusion strategy being less than or equal to the total capacity of the on-chip cache of the AI chip, determining the intermediate inter-layer fusion strategy as the target inter-layer fusion strategy.

7. The method of claim 2, wherein, The method comprises: Will The candidate inter-layer fusion strategy with the lowest target exchange volume among the candidate inter-layer fusion strategies is determined as the intermediate inter-layer fusion strategy. in response to the peak inference memory consumption of the intermediate inter-layer fusion strategy being greater than the total capacity of the on-chip cache of the AI chip, deleting the intermediate inter-layer fusion strategy from the at least two candidate inter-layer fusion strategies.

8. The method of claim 6, wherein, In a case where the neural network does not include a padding operation pad, the peak inference memory consumption is equal to a sum of a boundary occupancy, an input quantity of a peak layer, and an output quantity of the peak layer, the boundary occupancy is a sum of a boundary occupancy of each layer of the intermediate inter-layer fusion strategy, and the peak layer is a network layer in the intermediate inter-layer fusion strategy with a maximum sum of an input quantity and an input quantity in the target dimension.

9. The method of claim 8, wherein, In a case where the neural network includes the padding operation pad, the peak inference memory consumption is equal to a sum of a padding occupancy, the boundary occupancy, the input quantity of the peak layer, and the output quantity of the peak layer.

10. The method of claim 8 or 9, wherein, in a case where the network layer is a convolution layer and a corresponding convolution kernel is p-dimensional and a corresponding step length is q, the boundary occupancy of the network layer is p-q.

11. The method according to any one of claims 2 to 9, characterized in that, in a case where the peak inference memory consumption is equal to or greater than the total capacity of the on-chip cache of the AI chip and the neural network includes a down-sampling operation, the method further comprises: If the inference memory consumption peak of the neural network fused according to the candidate inter-layer fusion strategy corresponding to the layer block number 1 is less than the total on-chip cache capacity of the AI chip, a data combination layer is determined from the network layers of the neural network, the data combination layer processes input data to obtain output data when the obtained input data is equal to an input upper limit; wherein the sequence number of the data combination layer is greater than or equal to the sequence number of the network layer corresponding to the downsampling operation; In response to the inter-layer fusion strategy using the data combination layer satisfying the total on-chip cache capacity of the AI chip, the inter-layer fusion strategy using the data combination layer is determined as the target inter-layer fusion strategy.

12. The method according to any one of claims 2 to 9, characterized in that, The structure feature is used to indicate the connection relationship between at least two network layers of the neural network, and the input and output quantities of the network layers in a target dimension.

13. A data processing device for an artificial intelligence (AI) chip, characterized in that, The apparatus comprises: a feature acquisition module configured to acquire a structure feature of a neural network; a first determination module configured to determine at least two candidate inter-layer fusion strategies based on the structure feature, the candidate inter-layer fusion strategies being used to fuse network layers in the neural network; a second determination module configured to determine a target inter-layer fusion strategy from the at least two candidate inter-layer fusion strategies, the target inter-layer fusion strategy satisfying a preset condition in terms of a target exchange quantity and an inference memory consumption peak of the neural network after fusion, the target exchange quantity being used to indicate the data exchange quantity between an on-chip cache and an off-chip storage of the AI chip in a target dimension in a single inference process, the inference memory consumption peak being used to indicate the maximum value of the on-chip cache consumed by the neural network in an inference process, and the preset condition including that the inference memory consumption peak is less than the total on-chip cache capacity of the AI chip, and the target exchange quantity is the minimum value among the candidate inter-layer fusion strategies for inference on the AI chip; a network fusion module configured to fuse adjacent network layers in the neural network based on the target inter-layer fusion strategy to obtain a fused neural network; a data processing module configured to process to-be-inferred data based on the fused neural network.

14. An artificial intelligence, AI, chip, wherein The AI chip is provided with the data processing apparatus of claim 13.

15. A computer device, comprising: The computer device comprises a central processing unit (CPU) chip, an AI chip, and a memory, and the CPU chip, the AI chip, and the memory are connected through a bus; The AI chip comprises the data processing apparatus of claim 13.

Images