Data processing device, data processing method and related device

Abstract

A data processing device, a data processing method and a related device are applied to the technical field of computers. The data processing device comprises an addition operation tree, the addition operation tree comprises a plurality of adders, and each adder in the plurality of adders is a node of the addition operation tree. Wherein leaf nodes of the additive operation tree are used for receiving multiple pieces of data to be operated, each of the multiple adders is used for executing additive operation on the input data, input of a father node of the multiple adders is an operation result of a child node of the father node, and at least one node of the additive operation tree is used for outputting the operation result. When the accumulation operation is executed based on the addition operation tree, an intermediate result of the accumulation operation does not need to be stored and reloaded midway, so that the data read-write frequency in the accumulation operation process is greatly reduced, and the execution time of the accumulation operation is shortened.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing device, a data processing method, and related devices. Background Art

[0002] With the continuous development of chip technology, the number of electronic components that can be integrated per unit area of ​​a chip continues to increase, and the computing power of chips continues to rise. However, with the widespread application of technologies such as big data and artificial intelligence, the algorithms developed to process more data and solve more complex problems are becoming increasingly complex. As a result, the time consumed by algorithms running on chips continues to increase at an alarming rate, and the algorithm's time performance has gradually become a bottleneck in its application.

[0003] Among them, multi-data accumulation operation is an important reason for the low time performance of the algorithm. Multi-data accumulation refers to the operation of adding multiple data together. For example, some artificial intelligence (AI) algorithms include Transformer modules, and the Softmax operator of the Transformer module includes multi-data accumulation operation. Since the number of adders designed in the chip is limited, during multi-data accumulation operation, the adder can only perform one round of addition operation at a time. Figure 1 The chip is designed with four parallel adders. For example, to calculate the cumulative result of eight data points (A1 to A8), in the first round of calculation, the chip reads the eight data points A1 to A8 and performs pairwise addition operations on them, resulting in four addition results (denoted as B1 to B4). Since the accumulation calculation has not yet completed, B1 to B4 must be stored. In the second round of calculation, the chip reads the four data points B1 to B4 and performs pairwise addition operations on them again, resulting in two addition results (denoted as C1 and C2). Since the accumulation calculation has not yet completed, C1 and C2 still need to be stored. In this second round of calculation, since the input data is reduced to four, only two adders are used, and the remaining adders are idle. Similarly, until the final cumulative result is obtained, the results of each round of addition must be stored and read back into the adder input for addition in the next round of calculation. When using such a chip to run an algorithm involving multiple data accumulation operations, these read, compute, and store operations must be performed multiple times, and the amount of data to be accumulated is often large. On the one hand, the frequent data read and write during the accumulation process takes a long time to execute, resulting in poor timing performance of the algorithm. On the other hand, as the number of calculation rounds increases, more and more adders become idle, resulting in wasted computing resources and reduced chip energy efficiency.

[0004] Therefore, how to optimize the multi-data accumulation calculation process, improve the algorithm time performance and improve the energy efficiency of the chip has become an urgent problem that needs to be solved. Summary of the Invention

[0005] This application provides a data processing device, data processing method, and related devices that can optimize the multi-data accumulation process, significantly reduce the number of data reads and writes during the accumulation process, and improve the algorithm's time performance. When the data processing device is a chip or the data processing device is applied to a chip, the chip's energy efficiency can be greatly improved.

[0006] In a first aspect, the present application provides a data processing device comprising an addition operation tree, the addition operation tree comprising a plurality of adders, each of the plurality of adders being a node of the addition operation tree. Leaf nodes of the addition operation tree are configured to receive a plurality of data to be operated on, each of the plurality of adders is configured to perform an addition operation on the input data, the input of a parent node of the plurality of adders is a result of an operation on a child node of the parent node, and at least one node of the addition operation tree is configured to output the result of the operation.

[0007] In the present application, multiple adders are arranged in a tree structure. When performing an accumulation operation, the calculation results of the child nodes can be input into the parent node to continue the next round of addition operation, so that the intermediate results of the accumulation calculation do not need to be stored and re-read in the middle, which greatly reduces the number of data read and write times during the accumulation operation and shortens the execution time of the accumulation operation. Moreover, during the accumulation operation, as the number of rounds of addition calculations proceeds, the amount of data that needs to be added gradually decreases. In the tree structure, the number of nodes in each layer also gradually decreases towards the root node, and the calculation results of the child nodes can flow to the parent node to continue the calculation. Therefore, the tree structure can perfectly adapt to the operation process of the accumulation operation. Along the direction from the leaf node to the root node, the accumulation calculation is smoothly completed in the form of a pipeline, greatly reducing the idle time of the adder during the execution of the accumulation operation.

[0008] In summary, using this data processing device to handle the accumulation operation in the algorithm can significantly improve the algorithm's time performance. The data processing device is relatively energy efficient when performing accumulation operations. When the data processing device is a chip or the data processing device is applied to a chip, it can greatly improve the chip's energy efficiency.

[0009] In one possible implementation of the first aspect, the at least one node comprises a root node of an addition operation tree. The root node is used to output the final operation result of the addition operation tree. The root node is the endpoint of the data flow for all nodes in the tree except itself. Therefore, the output of the root node is the cumulative result of all data input into the addition operation tree, that is, the cumulative result of multiple data.

[0010] Optionally, only the root node in the addition operation tree can output the operation result to the outside of the addition operation tree. That is, the final operation result of the addition operation tree is uniformly output from the root node. In this case, the output of the root node can be designed to obtain the cumulative result of multiple data, which helps simplify the wiring design of the data processing device.

[0011] In another possible implementation of the first aspect, non-root nodes can also output operation results. On the one hand, when the amount of data to be accumulated is less than the maximum amount of data that can be received by the addition operation tree, outputting the operation results from the intermediate nodes can reduce the distance the data flows, reduce the amount of calculation, and further shorten the execution time. On the other hand, the output of the intermediate nodes can obtain an intermediate result of the accumulation operation. This intermediate result can be used in certain algorithms that require intermediate results, thereby increasing the diversity of data available to the addition operation tree and enhancing its usability.

[0012] In another possible implementation of the first aspect, the data processing device further includes a selector configured to receive an output of at least one node of the addition operation tree and determine a final operation result of the addition operation tree based on the output of the at least one node. This can improve the flexibility and usability of the addition operation.

[0013] In yet another possible implementation of the first aspect, the data processing device further includes a data transmission line.

[0014] Optionally, the addition operation tree is arranged on a data transmission line. Furthermore, the direction from the leaf nodes of the addition operation tree to the root node of the addition operation tree is the same as the transmission direction of the data to be operated. Because the tree structure has a natural data flow direction, it can adapt to the data flow direction of the data transmission line. Therefore, the addition operation tree can be designed as an on-path unit, that is, it can perform accumulation operations on multiple data during the data transmission process, which can significantly shorten the execution time of the accumulation operation.

[0015] In another possible implementation of the first aspect, the data processing device further includes a computing unit. The aforementioned data transmission line is a line for sending data to the computing unit. That is, an accumulation operation can be performed while data is being transmitted to the computing unit. In this case, the computing unit can directly use the accumulated result of the data without having to re-accumulate, thereby improving the algorithm's time performance.

[0016] In another possible implementation of the first aspect, the data transmission line is a line for writing data to the storage unit. That is, after the data calculation is completed, the data can be accumulated during the data writing process, and the subsequent calculation process can directly obtain the accumulated data result, thereby improving the time performance of the algorithm.

[0017] Furthermore, the storage unit may be an internal storage unit of the data processing device. Exemplarily, the data processing device further includes a storage unit. Alternatively, the storage unit is a storage unit other than the data storage unit. In this case, the data processing device is connected to the storage unit. This connection includes direct and / or indirect connection. Indirect connection, for example, is connected through another device. For example, a computing chip may be connected to a memory through a storage controller.

[0018] In another possible implementation of the first aspect, a connection path between at least one child node and a parent node of at least one child node in the addition operation tree can be controlled to be on or off. This on / off control can further enhance the flexibility of the addition operation tree and reduce energy consumption of the data processing device.

[0019] In another possible implementation of the first aspect, the logical storage location of the final operation result of the addition operation tree is adjacent to the logical storage locations of the multiple data to be operated. In this way, the operation result of the data can be stored adjacent to the data for easy access.

[0020] In another possible implementation of the first aspect, there is an offset between the logical storage location of the final operation result of the addition operation tree and the logical storage locations of the multiple data to be operated, which can improve the flexibility of storage control.

[0021] Optionally, the offset may be predefined, calculated, or obtained according to a predefined rule.

[0022] In another possible implementation of the first aspect, the logical storage location of the final operation result of the addition operation tree and / or the logical storage locations of the plurality of data to be operated on are obtained by application. For example, the storage control module allocates storage addresses for the final operation result of the addition operation tree and / or the plurality of data to be operated on.

[0023] In another possible implementation of the first aspect, a leaf node of the addition operation tree is used to receive one or two data among the multiple data. For example, a leaf node can receive at most two data.

[0024] In another possible implementation of the first aspect, the leaf nodes of the addition operation tree have corresponding serial numbers. When the number of multiple data to be calculated is less than the maximum number of data that can be received by the addition operation tree, the multiple data to be calculated are input into some leaf nodes in the order of the serial numbers of the leaf nodes of the addition operation tree or in reverse order. In this way, when the amount of data is relatively small, it can be fixedly allocated to some nodes for calculation, which facilitates on-off control. For example, when the amount of data is small, some adders can be controlled to be in a non-working state to save energy.

[0025] In another possible implementation of the first aspect, when the number of the multiple data to be calculated is less than the maximum number of receivable data of the addition operation tree, the multiple data to be calculated are randomly input into the leaf nodes of the addition operation tree.

[0026] When the amount of data is relatively small, it can be flexibly distributed to each adder, so that each adder is used evenly, slowing down the aging of electronic components related to the addition operation tree.

[0027] In another possible implementation of the first aspect, the data processing device is a chip. Optionally, the chip is an artificial intelligence chip, which is used to run artificial intelligence algorithms. For example, the data processing device is a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0028] In a second aspect, the present application provides a data processing method, which includes: obtaining multiple data and a first instruction, performing cumulative calculation on the multiple data through an addition operation tree, and obtaining a calculation result output by at least one node of the addition operation tree. The first instruction indicates to perform cumulative calculation on the multiple data. The addition operation tree includes multiple adders, each of the multiple adders is a node of the addition operation tree, and the leaf node of the addition operation tree is used to receive multiple data to be calculated. Each adder in the multiple adders is used to perform an addition operation on the input data, and the input of the parent node in the multiple adders is the calculation result of the child node of the parent node.

[0029] In a possible implementation of the second aspect, the at least one node includes a root node of an addition operation tree. The root node is used to output a final operation result of the addition operation tree.

[0030] In another possible implementation of the second aspect, the method further includes: determining a final operation result according to an output of at least one node.

[0031] In another possible implementation of the second aspect, the method further includes: obtaining a gating indication, where the gating indication is used to control the on-off of a connection path between at least one child node in the addition operation tree and a parent node of the at least one child node.

[0032] In another possible implementation of the second aspect, the selection indication includes a mask of multiple data. For example, when calculating the cumulative result of 8 data, the data transmitted in the data transmission line is: 0x 0102 0304 0506 0708F0. Here, F0 is a mask, and each bit of the mask is used to indicate whether a leaf node is connected or disconnected. Since F0 is 1111 0000, the mask indicates that, of the 8 leaf nodes in the deepest layer, the first 4 leaf nodes are connected, and the last 4 leaf nodes are disconnected.

[0033] In a third aspect, the present application provides a chip system comprising a computing chip, a memory controller, and a memory. The computing chip is used to perform computing functions, and the memory controller is connected to the memory. The memory controller is used to store the computing results of the computing chip in the memory via a data transmission line and / or read data from the memory for use by the computing chip. The computing chip and / or the memory controller include the data processing device described in the first aspect or any possible embodiment of the first aspect.

[0034] Optionally, the computing chip is an artificial intelligence chip, which is used to run artificial intelligence algorithms.

[0035] In a fourth aspect, the present application provides a terminal comprising the data processing device described in the first aspect or any possible embodiment of the first aspect, or comprising the chip system described in the third aspect. Optionally, the terminal comprises an intelligent terminal or transportation vehicle such as a vehicle, robot, drone, or ship.

[0036] In a fifth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed by a computing device, they are used to implement the method described in the second aspect or any possible implementation method of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The following is a brief introduction to the drawings required for describing the embodiments.

[0038] Figure 1 It is a schematic diagram of a parallel arrangement of adders performing accumulation operations;

[0039] Figure 2 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0040] Figure 3 This is a schematic diagram of the operation process of an addition operation tree provided in an embodiment of the present application;

[0041] Figure 4 This is a schematic diagram of an output method of an addition operation tree provided in an embodiment of the present application;

[0042] Figure 5 This is a schematic diagram of another output method of an addition operation tree provided in an embodiment of the present application;

[0043] Figure 6 This is a schematic diagram of an output method of an addition operation tree provided in an embodiment of the present application;

[0044] Figure 7 This is a schematic diagram of the structure of an addition operation tree provided in an embodiment of the present application;

[0045] Figure 8 This is a schematic diagram of the operation process of another addition operation tree provided in an embodiment of the present application;

[0046] Figure 9 This is a schematic diagram of the operation process of another addition operation tree provided in an embodiment of the present application;

[0047] Figure 10 This is a structural diagram of another data processing device provided in an embodiment of the present application;

[0048] Figure 11 This is a structural diagram of another data processing device provided in an embodiment of the present application;

[0049] Figure 12 This is a schematic diagram of the structure of a chip system provided in an embodiment of the present application;

[0050] Figure 13 It is a flowchart of a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] AI has become an indispensable technology in our daily lives. The essence of AI is to train machines to think like the human brain. Developers typically feed large amounts of data into machines, which then learn through specific algorithms, creating a model that describes the patterns between input stimuli and specific labels. Once trained, this model can be deployed on devices, making them as intelligent as humans.

[0052] In practical applications, the parameters of trained AI models need to be stored in hardware. At runtime, specific algorithms are used to process the model parameters and input data to produce the corresponding results. With the development of AI technology, the amount of data required for AI model parameters is growing, and AI algorithms are becoming increasingly complex, posing a severe challenge to the computing performance of AI hardware implementations.

[0053] For example, models including Transformer modules have been widely used in many fields in recent years, but their time performance on chips is far from satisfactory. One of the key bottlenecks is the implementation of the Softmax operator on the chip. An important operation of the Softmax operator is to accumulate multiple data. The addition unit in current AI chips is designed as multiple parallel adders, such as Figure 1 . If such an addition unit is used to perform multi-data accumulation operations, before the final accumulation result is obtained, the results of each round of addition operations must be temporarily stored in the storage unit and re-read to the input end of the adder for the next round of addition operations. Such reading, calculation, and storage operations need to be performed multiple times, causing data to be read and written frequently. When the amount of data is large, it usually takes more cycles to complete the calculation. Moreover, as the operation continues, the data loaded into the input end of the adder becomes less and less, causing some adders to be idle, and the degree of parallelism will become lower and lower during the operation.

[0054] In view of this, the present application provides a data processing device, data processing method, and related devices that can optimize the multi-data accumulation operation process, significantly reduce the number of data reads and writes during the accumulation operation, and improve the algorithm's time performance. When the data processing device is a chip or the data processing device is applied to a chip, the chip's energy efficiency can be greatly improved.

[0055] The following first introduces the data processing device provided in the embodiment of the present application.

[0056] See Figure 2 , Figure 2 1 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Data processing device 10 includes an addition operation tree 11, which includes multiple adders 111. Multiple adders 111 are arranged in a tree structure, and each adder 111 can serve as a node in addition operation tree 11. It should be noted that this application uses a tree with 15 adders in 4 layers as an example. In specific implementations, the number and arrangement of adders in the addition operation tree may be designed differently.

[0057] In some solutions, the addition operation tree 11 is a binary tree. In a binary tree, each node can have at most two subtrees. Figure 2 As shown, multiple adders are arranged to form 4 layers. The first layer includes 1 adder (considered as the root node), the second layer includes 2 adders, the third layer includes 4 adders, and the fourth layer includes 8 adders. In the tree structure, there is a correlation between the nodes. For example, the adder in the first layer is the parent node of the two adders in the second layer. Correspondingly, the two adders in the second layer are the child nodes of the adder in the first layer. Of course, Figure 2The number and arrangement of adders shown are only examples. In some embodiments, the number of adders may be more or less, and the number of adders included in each layer may also have other designs.

[0058] During the accumulation operation, the leaf nodes of the addition operation tree 11 are used to receive multiple data to be operated, wherein the leaf node refers to a node without child nodes, and the input of the leaf node is the data to be operated. A leaf node can receive 1 or 2 data inputs. For example, combined with Figure 3 For example, an addition operation tree performs an accumulation calculation on 16 data. The 16 data can be represented as A1 to A16. These 16 data can be input into the 8 adders on the 4th layer of the addition operation tree 11. Each adder at a leaf node position receives 2 data. For example, A1 and A2 can be input into adder 111a as a leaf node, and A3 and A4 can be input into adder 111b as a leaf node. The same applies to other cases. Of course, the binary tree is used as an example here. For other tree structures, the leaf nodes can receive more or less data.

[0059] Each adder in the addition operation tree 11 is used to perform an addition operation on input data. For example, adder 111a performs an addition operation on A1 and A2, and the result obtained is B1. Similarly, adder 111b performs an addition operation on A3 and A4, and the result obtained is B2.

[0060] In multiple adders, the input of the parent node is the operation result of the child node of the parent node. Figure 3 Adders 111a and 111b are child nodes of adder 111c. The inputs to adder 111c are the operation results of adder 111a and adder 111b, that is, the inputs to adder 111c are B1 and B2. Adder 111c performs an addition operation on B1 and B2, and the result obtained is C1. Furthermore, adders 111c and 111d are child nodes of adder 111e. Therefore, adder 111e can perform an addition operation on C1 and C2, and obtain a result D1. Similarly, adder 111f can add the operation results D1 and D2, and obtain a result E1.

[0061] In the addition operation tree 11, the operation results of multiple data can be output from at least one node to the outside of the addition operation tree 11. It should be understood that in this article, the operation results of multiple data (i.e., the operation results output by the addition operation tree) include the final result of the accumulation operation (such as E1) and optionally also include the intermediate results of the accumulation operation (such as B1, C1, D1, etc.). For ease of understanding, two possible ways of outputting the operation results are introduced below:

[0062] Method 1: The operation results of multiple data are output from the root node of the addition operation tree 11. The root node is the data flow endpoint of all nodes in the tree except itself, so the output of the root node is the cumulative result of all data input into the addition operation tree 11, that is, the final result of the cumulative operation. Figure 3 As shown, the output E1 of the root node (adder 111f) is the accumulated result of A1 to A16.

[0063] In one possible implementation, only the root node in the addition operation tree 11 can output the operation result to the outside of the addition operation tree. That is, the operation results of multiple data are uniformly output from the root node. For example, Figure 3 In the addition operation tree 11 shown, adders at non-root nodes, such as adder 111a and adder 111c, do not output operation results to the outside of the adder.

[0064] In the second method, the operation results of multiple data can be output from non-root nodes. Figure 4 As shown, one or more non-root nodes such as adder 111a and adder 111c can also output operation results. Operation results such as B1, C1, and B2 are intermediate results of the accumulation operation. The output method shown in method 2 can be applied to various scenarios. Two exemplary application scenarios are listed below:

[0065] Scenario 1: The number of data to be accumulated is less than the maximum number of data that can be received by the addition operation tree 11. In this case, outputting the operation results from the nodes in the middle layer can reduce the distance that the data flows through, reduce the amount of operation and further shorten the execution time. Figure 4 For example, the addition operation tree 11 includes 8 leaf nodes, and the maximum number of data that can be received is 16. Consider a possible situation. If the addition operation tree 11 is used to calculate the accumulation of 4 data, taking the input data A1, A2, A3 and A4 as an example, the operation result C1 is the accumulation result of A1, A2, A3 and A4. Therefore, the output of the adder 111c can be used as the final operation result without going through subsequent adder operations.

[0066] Furthermore, when using the addition operation tree 11, the user or machine can determine the leaf nodes where the data to be accumulated are input, and based on the positions of these leaf nodes in the addition operation tree 11, select which node's output is used as the final operation result.

[0067] In scenario 2, the algorithm requires both the final result of the cumulative operation and the intermediate results of the cumulative operation. Intermediate results include the result of adding two of them together (such as B1 and B2), or the result of adding four data in the same group (such as C1 and C2), or the result of adding eight data in the same group (such as D1 and D2). Optionally, the data in the same group can be input into the same subtree. In this case, the intermediate results of the cumulative operation can be obtained through the output of the non-root node.

[0068] In some possible implementations, the data processing device 10 further includes a selector 13, such as Figure 4 As shown, the selector 13 is used to receive the output of at least one node of the addition operation tree and determine the final operation result of the addition operation tree 11 based on the output of the at least one node. Optionally, the final operation result of the addition operation tree 11 may include the final result and / or intermediate results of the accumulation operation. The specific selection method can be designed according to needs.

[0069] For example, the selector 13 can select one or more of B1, C1, B2, etc. as the final operation result of the addition operation tree 11. Exemplarily, when the data to be operated is A1-A16, if the data required by the algorithm is the final result of the accumulation operation, the selector 13 can select E1 as the final operation result. As another example, when the data to be operated is A1-A16, if the data required by the algorithm is the accumulation result of all the data and the result of adding 8 data in the same group, the selector 13 can select E1, D1 and D2 as the final operation results, where E1 is the final result of the accumulation operation, D1 is the result of adding 8 data in one group (A1 to A8), and D2 is the result of adding 8 data in another group (A9 to A16). In this way, the flexibility and usability of the addition operation tree 11 can be improved.

[0070] As can be seen from the above description of the data processing device 10, when the data processing device 10 is used to perform an accumulation operation, the calculation results of the child nodes can be input into the parent node to continue the next round of addition operation, so that the intermediate results of the accumulation calculation do not need to be stored and re-read midway, which greatly reduces the number of data reads and writes during the accumulation operation and shortens the execution time of the accumulation operation. Moreover, during the accumulation operation, as the number of rounds of addition calculations progresses, the amount of data that needs to be added gradually decreases. In the tree structure, the number of nodes in each layer also gradually decreases towards the root node, and the calculation results of the child nodes can flow to the parent node to continue the calculation. Therefore, the tree structure can perfectly adapt to the operation process of the accumulation operation. Along the direction from the leaf node to the root node, the accumulation calculation is smoothly completed in the form of a pipeline, greatly reducing the idle time of the adder during the accumulation operation.

[0071] Especially for AI-related algorithms, this application greatly improves the performance of Softmax through a tree-structured adder, thereby achieving the goal of improving the performance of Transformer-type neural networks. Of course, in addition to Transformer-type neural networks, other algorithms involving accumulation operations can also be accelerated by the data processing device provided by this application.

[0072] The following is an introduction to some possible designs of the data processing device.

[0073] In some possible implementations, the connection path between at least one child node and at least one of its parent nodes in the addition operation tree 11 can be controlled to be on or off. By controlling the on or off state, the data flow between nodes can be controlled, thereby controlling the accumulation operation process and improving the flexibility and usability of the addition operation tree.

[0074] See Figure 5 In the addition operation tree 11, the connection path between the leaf node and its parent node can be controlled to be on and off. As a possible implementation, a switch 112 can be set between the leaf node and the parent node of the addition operation tree 11. The switch 112 is used to control whether the operation result of the leaf node can be input into the parent node.

[0075] In some scenarios, the amount of data that needs to be accumulated is less than the maximum amount of data that can be received by the addition operation tree. In this case, the connection path between the leaf node without data input and the parent node of the control part is disconnected, which can reduce the amount of calculation of the data processing device and avoid the influence of noise. Figure 5 When only adder 111a and adder 111b in the addition operation tree 11 receive input data, the connection path between adder 111a and adder 111c and the connection path between adder 111a and adder 111c are in a pass state, so that B1 and B2 can input adder 111c, while the connection paths between the remaining leaf nodes and the parent node are disconnected.

[0076] In other scenarios, it's necessary to remove some of the centrally stored data during the accumulation calculation. By controlling the connection path between a node and its parent node, some of the input data can be removed during the accumulation calculation, resulting in the accumulation result for only the remaining portion of the data. For example, if A1 and A2 are removed from the inputs A1 to A16 and are not included in the accumulation process, the connection path between adders 111a and 111c can be disconnected.

[0077] It should be understood that Figure 5In the embodiment shown, only the connection path between the leaf node and its parent node is shown to be controllable. In a specific implementation, other positions in the addition operation tree can also be set to a controllable on-off state. Figure 6 In some solutions, one or more layers in layers 1 to 4 of the addition operation tree 11 can be controlled to be on and off. In other solutions, the data input layer, that is, the path input to the leaf node, can also be controlled to be on and off.

[0078] In some possible implementations, the on / off control of the connection paths in the addition operation tree 11 can be implemented using a data mask. For example, the data transmitted on the data line connecting the addition operation tree is: 0x 0102030405060708F0. In 0x0102 0304 0506 0708, each 8 bits represents a piece of data to be operated on, that is, the data to be operated on are 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, and 0x08, and F0, that is, 1111 0000, is the data mask. Each bit of the mask is used to indicate the on / off status of a leaf node. Therefore, 1111 0000 means that among the 8 leaf nodes in the deepest layer (i.e., the 4th layer), the connection paths between the first 4 leaf nodes and the parent node are connected, and the connection paths between the last 4 leaf nodes and the parent node are disconnected, and the 8 data to be calculated can be input into the first 4 leaf nodes for calculation.

[0079] The above text has mentioned several times the design related to the number of data input to the addition operation tree 11. The following introduces two possible operation implementations of the addition operation tree when the number of multiple data to be operated is less than the maximum number of data that the addition operation tree 11 can receive.

[0080] In operation implementation 1, the leaf nodes of the addition operation tree have corresponding sequence numbers. When the number of data to be operated on is less than the maximum number of data that the addition operation tree can accept, the data to be operated on are input into some leaf nodes in the order or reverse order of the sequence numbers of the leaf nodes of the addition operation tree. It should be noted that the sequence numbers here are intended to indicate that some leaf nodes are fixed and preferentially used for addition operations. In actual operations, leaf nodes may not actually have corresponding sequence numbers.

[0081] See Figure 7 The maximum number of data that the addition operation tree can receive is 16. If the addition operation tree is used to calculate the accumulation of 4 data (A1, A2, A3 and A4), these 4 data can be sequentially input into the first leaf node and the second leaf node, namely adder 111a and adder 111b.

[0082] Combined with some of the designs above, the final operation results of A1, A2, A3 and A4 can be output from adder 111c without entering the subsequent adder. Furthermore, the connection paths through which data flows are in a connected state, while the connection paths through which data does not flow can be controlled to be disconnected. Alternatively, the data transmitted by the connection paths through which data does not flow can be set to 0. Of course, Figure 7 The implementation shown is only an example. In a specific embodiment, the final operation results of A1, A2, A3 and A4 may also be output from the root node of the addition operation tree, and / or no on-off control may be performed between the connection lines.

[0083] In operation implementation 2, when the number of data to be calculated is less than the maximum number of data that can be received by the addition operation tree, the data to be calculated are flexibly arranged to the leaf nodes of the addition operation tree, without having to follow a fixed order. For example, the data to be calculated are randomly input into the leaf nodes of the addition operation tree.

[0084] See Figure 8 The maximum number of data that the addition operation tree can receive is 16. If the addition operation tree is used to calculate the accumulation of 4 data (A1, A2, A3 and A4), these 4 data can be flexibly input into the leaf nodes of the addition operation tree. For example, A1 and A2 are input into adder 111a, while A3 is input into adder 111g, and A4 is input into adder 111h.

[0085] Combined with some of the designs above, the final operation results of A1, A2, A3 and A4 can be output from adder 111e without entering the subsequent adder. Furthermore, the connection paths through which data flows are in a connected state, while the connection paths through which data does not flow can be controlled to be disconnected. Alternatively, the data transmitted by the connection paths through which data does not flow can be set to 0. Of course, Figure 8 The implementation shown is only an example. In a specific embodiment, the final operation results of A1, A2, A3 and A4 may also be output from the root node of the addition operation tree, and / or no on-off control may be performed between the connection lines.

[0086] In some scenarios, the final operation result of the addition operation tree can be stored adjacent to the storage locations of the multiple data of the input addition operands, or it can be stored elsewhere. The following describes these two cases by way of example.

[0087] In case 1, the logical storage location of the final operation result is adjacent to the logical storage locations of the multiple data to be operated. As a possible example, the addition operation tree is set on the data write path. For example, the data to be written to the storage unit includes 0x0102030405060708. In 0x0102030405060708, each 8 bits represents a data to be operated, that is, the data to be operated are 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 and 0x08. The addition operation can be performed in the data write path to obtain the cumulative result of the aforementioned 8 data, 0x24 (here using hexadecimal counting as an example). At this time, the data written to the storage unit can be 0x010203040506070824. 0x24 is the sum of 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, and 0x08. It can be seen that 0x 01 02 03 04 05 06 07 08 and 0x24 are stored in adjacent locations.

[0088] This example uses the case where the output of an addition operation tree is the cumulative result of multiple data items. If the output of the addition operation tree also includes intermediate results, the multiple data items to be operated on, the cumulative result of the multiple data items, and the intermediate results can be stored adjacent to each other. Alternatively, the cumulative result of the multiple data items can be stored adjacent to the multiple data items to be operated on, while the intermediate results can be stored elsewhere.

[0089] In some implementations, after performing the accumulation operation on multiple data items, the data items themselves are not discarded. Instead, their values ​​are retained and transmitted along with the output of the addition operation tree. For example, after performing the accumulation operation on 0x 0102 0304 05060708, the data written as 0x010203 0405 0607 0824 retains the original values.

[0090] Case 2: There is an offset between the logical storage location of the final operation result and the logical storage location of the multiple data to be calculated. Optionally, the offset can be predefined, calculated, or obtained according to a predefined rule. For example, the logical storage location of the final operation result and / or the logical storage location of the multiple data to be calculated is calculated according to the idle state of the current storage space. Another example is randomly determining the offset between the logical storage location of the final operation result and the logical storage location of the multiple data to be calculated. As a possible example, the addition operation tree is set on the path of writing data. For example, the data that needs to be written to the storage unit includes 0x 0102 0304 0506 0708, and the address to which the data is written is 0x1000. An addition operation can be performed on the path of writing data to obtain the cumulative result 0x24 of the aforementioned 8 data, and the address to which the cumulative result 0x24 is written is "0x1000+first offset".

[0091] Optionally, when data needs to be written to an internal buffer, the two addresses here are both addresses of the internal buffer. When data needs to be written to external storage (such as a double data rate synchronous dynamic random access memory), the two addresses here are both addresses of the external storage.

[0092] In case 3, the logical storage location of the final operation result of the addition operation tree and / or the logical storage location of the multiple data to be operated are obtained by application. For example, the storage control module allocates storage addresses for the final operation result of the addition operation tree and / or the multiple data to be operated.

[0093] As mentioned above, the addition operation tree 11 can be set on the data transmission path. Some related implementation methods are introduced below.

[0094] For some possible implementations, see Figure 2 The data processing device 10 further includes a data transmission line 12, on which the addition operation tree 11 is disposed. Furthermore, the direction from the leaf nodes of the addition operation tree 11 to the root node of the addition operation tree 11 is the same as the transmission direction of the data to be operated on. Since the tree structure has a natural data flow direction, it can be adapted to the data flow direction of the data transmission line.

[0095] Therefore, the addition operation tree can be designed as an on-path unit, that is, it can perform accumulation operations on multiple data as the data is transmitted, which can significantly shorten the execution time of the accumulation operation. For example, the data with the operation is transmitted from the data transmission line, and as the data flows from left to right, it passes through each node in the addition operation tree in the direction of flow, thus completing the accumulation process. The accumulation result output from the addition operation tree continues to flow along the data transmission line, so that the accumulation result can be used by the calculation unit and / or stored.

[0096] In some other possible implementations, see Figure 9 The data processing device 10 further includes a computing unit 14, and the data transmission line 12 is a line for sending data to the computing unit 14. That is, during the process of transmitting multiple data to the computing unit 14, these multiple data can be accumulated through the addition operation tree, so that the computing unit 14 can obtain the accumulated results of the multiple data (optionally including intermediate results). This eliminates the need to repeatedly read and write data to the storage unit, eliminates the need to consume additional time for accumulation operations, and shortens the execution time of the accumulation operation.

[0097] For some possible implementations, see Figure 10 The data processing device 10 further includes a storage unit 15, and the data transmission line 12 is a line for writing data to the storage unit 15. Specifically, when writing multiple data items to the storage unit 15, these data items can be accumulated by the addition operation tree, so that the accumulated result of the multiple data items can be stored in the storage unit 15 for use in subsequent calculation processes. The storage location of the accumulated result can be found in the previous description.

[0098] In some other possible implementations, the data transmission line 12 is a line for reading data from the storage unit 15. That is, in the process of reading a plurality of data, an accumulation operation is performed along the line to obtain an accumulation result of the plurality of data.

[0099] Alternatively, the aforementioned storage unit 15 may be disposed outside the data processing device 10, in which case the data processing device 10 is connected to the storage unit 15. For example, the storage unit 15 is a double data rate synchronous dynamic random access memory (DDR SDRAM), also referred to as DDR by those skilled in the art.

[0100] For some possible implementations, see Figure 11 The data processing device 10 may include an acquisition unit 16 and a control unit 17. The acquisition unit 16 is configured to perform one or more operations such as receiving, transmitting, acquiring, inputting, and outputting. For example, the acquisition unit 16 may be one or more of a communication interface and an input / output interface. The control unit 17 is configured to control the operation and behavior of one or more modules in the data processing device. For example, it may control whether to enable the addition operation tree 11 and / or control the connection and disconnection of paths in the addition operation tree 11.

[0101] The aforementioned data processing device 10 can specifically be one or more computing devices, for example, computing devices include but are not limited to processors, controllers, and processors and / or chip systems, or computing platforms. Among them, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), or a digital signal processor (DSP); in another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, for example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The controller is a hardware or software device, which is usually connected to other devices and controls the operation and behavior of other devices through the connected lines. In some schemes, the controller can also be regarded as a processor that can execute program instructions and complete the functions corresponding to the instructions. In some implementations, multiple computing devices can be integrated together in the form of a system-on-chip (SOC). The SOC may include at least one processor, and the type of the at least one processor may be different, for example, including a CPU and an NPU.

[0102] The following is an exemplary introduction to a system in which the data processing device of the present application can be applied.

[0103] The present application also provides a chip system, including a computing chip, a storage controller and a memory. The computing chip is used to perform computing functions, the storage controller is connected to the memory, and the storage controller is used to store the calculation results of the computing chip in the memory through a data transmission line and / or read data from the memory for use by the computing chip. The computing chip and / or the storage controller include the aforementioned data processing device 10. Exemplarily, the computing chip is the data processing device 10, or the data processing device 10 is part of the computing chip. Again exemplarily, the storage controller is the data processing device 10, or the data processing device 10 is part of the storage controller.

[0104] See Figure 12 , Figure 12 This is a schematic diagram of the structure of a chip system provided by this application, which includes an NPU, a storage controller and a memory. For example, the memory is DDR. Among them, the NPU is a chip for performing deep learning calculations, that is, a computing chip. The NPU includes a computing unit, a buffer and a connecting line. The computing unit is used to perform calculations, including Figure 12 The tensor processing unit and vector processing unit are shown. The buffer provides storage space, and the connection lines are used to transmit data. The memory controller is a device for controlling access to memory and connects the NPU and DDR.

[0105] In one scenario, an addition operation tree 11 may be provided in the NPU. The addition operation tree 11 may be provided on a line for writing data to a buffer, or on a line for reading data from a buffer, or on a line for transmitting data to a computing unit, or on a line for transmitting data from a computing unit to another module. Of course, the functions of the aforementioned lines may be combined or overlapped. For example, the line for reading data from a buffer may also be the line for transmitting data to a computing unit.

[0106] Alternatively, the NPU may be regarded as the data processing device 10, or the data processing device 10 may be a part of the NPU. For example, the addition operation tree 11 may be included in an atomic add unit of the NPU, which is used to implement an accumulation operation.

[0107] In another scenario, an addition operation tree 11 may be provided in the storage controller. In this case, the data written by the storage controller to the memory may be written to the memory after passing through the addition operation tree 11, or the data read from the memory by the storage controller may be transmitted to the NPU after passing through the addition operation tree 11. Optionally, the storage controller may be regarded as the data processing device 10, or the data processing device 10 may be part of the storage controller. Exemplarily, the addition operation tree 11 may be included in the atomic add unit (atomic add) of the storage controller, which is used to implement the accumulation operation.

[0108] The present application also provides a terminal, which includes the aforementioned data processing device 10 or the aforementioned chip system.

[0109] The data processing method provided by this application is introduced below.

[0110] See Figure 13 , Figure 13 This is a flow chart of a data processing method provided by an embodiment of the present application. This method can be applied to the aforementioned data processing device, chip system and terminal, etc., for example, Figure 2 ,or Figure 11 For the convenience of description, the following is an introduction using the data processing device as the execution subject. The data processing device can be replaced by other devices, modules or equipment, such as chips, chip systems, etc. Figure 13 The data processing method shown may include steps S1301 and S1302. Steps S1301 and S1302 are specifically as follows:

[0111] Step S1301: The data processing device obtains a plurality of data and a first instruction.

[0112] The data processing device may be the aforementioned data processing device 10, or a device including a data processing device, such as an NPU, a storage controller, a chip system, etc. The data processing device includes an addition operation tree, and the structure of the addition operation tree can be seen in the description above, for example Figures 2 to 8 Related instructions.

[0113] The plurality of data refers to at least 2 data, such as 2, 4, 8, 16, etc. For example, the plurality of data is 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, and 0x08.

[0114] The first instruction is used to instruct to perform accumulation calculation on a plurality of data.

[0115] As one possible example, during a write process for multiple data items, if the data processing device receives a first instruction, indicating that an accumulation operation needs to be performed on the multiple data items, the data written to the storage unit by the data processing device includes the accumulation result of the multiple data items (and / or an intermediate result of the accumulation operation), and optionally also includes the multiple data items themselves. Conversely, if the data processing device does not receive the first instruction, or if the data processing device receives another instruction indicating that an accumulation operation is not required, the data processing device performs a normal write process on the multiple data items.

[0116] Similarly, in the process of transmitting multiple data to a computing unit, if the data processing device receives a first instruction, indicating that an accumulation operation needs to be performed on the multiple data, the data transmitted by the data processing device to the computing unit includes the accumulation result of the multiple data and / or the intermediate result of the accumulation operation, and optionally also includes the multiple data themselves. Conversely, if the first instruction is not received, or the data processing device receives another instruction indicating that an accumulation operation is not required, the data transmitted by the data processing device to the computing unit is the multiple data themselves.

[0117] Alternatively, the plurality of data and the first instruction may be obtained through different modules. For example, the first data may be obtained from an interface connected to the storage unit, or from an interface connected to the computing unit. The first instruction may be input from outside the data processing device.

[0118] Step S1302: The data processing device performs cumulative calculation on the multiple data through the addition operation tree to obtain a calculation result output by at least one node of the addition operation tree.

[0119] The addition operation tree includes multiple adders, each of which is a node of the addition operation tree. The leaf nodes of the addition operation tree are used to receive multiple data to be operated on. Each of the multiple adders is used to perform an addition operation on the input data. The input of the parent node of the multiple adders is the operation result of the child node of the parent node. For related description, please refer to the above.

[0120] Optionally, at least one node comprises a root node of an addition operation tree. Figure 3 、 Figure 4 Related instructions for other icons.

[0121] Alternatively, at least one node may include a root node of the addition operation tree and / or a non-root node of the addition operation tree. In this case, the data processing device may determine a final operation result based on the output of at least one node. As shown in FIG4 , the final operation result is determined by a selector from the outputs of B1, C1, B2, etc. The final operation result includes an accumulated result and / or an intermediate result of the accumulated operation.

[0122] In some scenarios, a connection path between at least one child node in an addition operation tree and a parent node of the at least one child node may be controlled to be on or off. As one possible implementation, a data processing device may obtain a gating indication, where the gating indication is used to control the connection path between the at least one child node in the addition operation tree and the parent node of the at least one child node.

[0123] For example, the strobe indication is a mask for multiple data. Figure 7 , for example, the multiple data are 0x 01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 and 0x08. The data obtained by the data processing device is 0x 01 0203 0405 060708F0, wherein each 8 bits in 0x 0102 0304 0506 0708 represent a data to be calculated. And F0 is a data mask, each bit of which is used to indicate the on / off of a leaf node. Therefore, F0, i.e. 1111 0000, means that among the 8 leaf nodes in the deepest layer (i.e. 4 layers), the connection path between the first 4 leaf nodes and the parent node is connected, and the connection path between the last 4 leaf nodes and the parent node is disconnected, and the 8 data to be calculated can be input into the first 4 leaf nodes for calculation.

[0124] exist Figure 13 In the illustrated embodiment, a data processing device can perform an accumulation operation on multiple data using an addition operation tree. Multiple adders are arranged in a tree structure. During the accumulation operation, the calculation results of the child nodes can be input into the parent node to continue the next round of addition operation. This eliminates the need to store and reread the intermediate results of the accumulation operation, greatly reducing the number of data reads and writes during the accumulation operation and shortening the execution time of the accumulation operation. Moreover, during the accumulation operation, as the number of rounds of addition calculations progresses, the amount of data that needs to be added gradually decreases. In the tree structure, the number of nodes in each layer also gradually decreases towards the root node, and the calculation results of the child nodes can flow to the parent node to continue the operation. Therefore, the tree structure can perfectly adapt to the operation process of the accumulation operation. From the leaf node to the root node, the accumulation operation is smoothly completed in the form of a pipeline, greatly reducing the idle time of the adder during the accumulation operation.

[0125] Especially for AI-related algorithms, this application greatly improves the performance of Softmax through a tree-structured adder, thereby achieving the goal of improving the performance of Transformer-type neural networks. Of course, in addition to Transformer-type neural networks, other algorithms involving accumulation operations can also be accelerated by the data processing device provided by this application.

[0126] The embodiment of the present application further provides a computer storage medium, wherein the computer storage medium includes computer instructions, and when the computer instructions are executed by a computing device, the computer storage medium is used to implement Figure 13 The data processing method shown.

[0127] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0128] The “at least one” mentioned in the embodiments of this application refers to one or more, and “plurality” refers to two or more. “At least one of the following items” or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. “And / or” describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character “ / ” generally indicates that the previous and next associated objects are in an “or” relationship.

Claims

1. A data processing device, characterized in that: The data processing device includes an addition operation tree, the addition operation tree includes a plurality of adders, and each adder in the plurality of adders is a node of the addition operation tree; The leaf nodes of the addition operation tree are used to receive multiple data to be operated; Each adder in the plurality of adders is used to perform an addition operation on input data, and the input of a parent node in the plurality of adders is an operation result of a child node of the parent node; At least one node of the addition operation tree is used to output an operation result.

2. The data processing device according to claim 1, wherein The at least one node includes a root node of the addition operation tree, and the root node is used to output a final operation result of the addition operation tree.

3. The data processing device according to claim 1, wherein: The data processing device further includes a selector, The selector is used to receive the output of at least one node of the addition operation tree, and determine the final operation result of the addition operation tree according to the output of the at least one node.

4. The data processing device according to any one of claims 1 to 3, characterized in that: The data processing device also includes a data transmission line, the addition operation tree is set on the data transmission line, and the direction from the leaf node of the addition operation tree to the root node of the addition operation tree is the same as the transmission direction of the data to be operated.

5. The data processing device according to claim 4, wherein: The data processing device further includes a computing unit, and the data transmission line is a line for sending data to the computing unit; or, The data transmission line is a line for writing data to a storage unit. The data processing device further includes the storage unit, or the data processing device is further connected to the storage unit.

6. The data processing device according to any one of claims 1 to 5, characterized in that: A connection path between at least one child node in the addition operation tree and a parent node of the at least one child node can be controlled to be on or off.

7. The data processing device according to claim 2 or 3, characterized in that The logical storage location of the final operation result is adjacent to the logical storage locations of the plurality of data to be operated.

8. The data processing device according to claim 2 or 3, characterized in that: There is an offset between the logical storage location of the final operation result and the logical storage locations of the plurality of data to be operated.

9. The data processing device according to any one of claims 1 to 8, characterized in that: A leaf node of the addition operation tree is used to receive one or two data among the multiple data.

10. The data processing device according to any one of claims 1 to 9, characterized in that: The leaf nodes of the addition operation tree have corresponding serial numbers. When the number of the multiple data to be calculated is less than the maximum receivable number of data of the addition operation tree, the multiple data to be calculated are input into some leaf nodes in the order of the leaf nodes of the addition operation tree or in reverse order.

11. The data processing device according to any one of claims 1 to 9, characterized in that: When the number of the plurality of data to be calculated is less than the maximum number of receivable data of the addition operation tree, the plurality of data to be calculated are randomly input into the leaf nodes of the addition operation tree.

12. A data processing method, characterized in that: The method comprises: Acquire a plurality of data and a first instruction, wherein the first instruction instructs to perform accumulation calculation on the plurality of data; Performing cumulative calculation on the plurality of data through an addition operation tree to obtain a calculation result output by at least one node of the addition operation tree; The addition operation tree includes multiple adders, each of the multiple adders is a node of the addition operation tree, the leaf node of the addition operation tree is used to receive multiple data to be operated, each of the multiple adders is used to perform addition operation on the input data, and the input of the parent node of the multiple adders is the operation result of the child node of the parent node.

13. The data processing method according to claim 12, characterized in that: The at least one node includes a root node of the addition operation tree, and the root node is used to output a final operation result of the addition operation tree.

14. The data processing method according to claim 13, wherein: The method further comprises: A final operation result is determined according to the output of the at least one node.

15. The data processing method according to any one of claims 12 to 14, characterized in that: The method further comprises: A gating indication is obtained, where the gating indication is used to control the on / off state of a connection path between at least one child node in the addition operation tree and a parent node of the at least one child node.

16. The data processing method according to claim 15, characterized in that: The strobe indication is a mask of the plurality of data.

17. A chip system, characterized in that: The chip system includes a computing chip, a storage controller and a memory, The computing chip is used to perform computing functions; The storage controller is connected to the memory, and is used to store the calculation results of the computing chip in the memory through a data transmission line and / or read data from the memory for use by the computing chip; The computing chip or the storage controller includes the data processing device according to any one of claims 1 to 11.

18. A terminal, characterized in that: The terminal includes the data processing device according to any one of claims 1 to 11, or includes the chip system according to claim 17.

19. A computer storage medium, characterized in that The computer storage medium includes computer instructions, which are used to implement the method according to any one of claims 12 to 16 when executed by a computing device.

Images