Compiling optimization method and device for computational graph, equipment and medium

Abstract

The invention discloses a compilation optimization method and device of a computational graph, equipment and a medium. The method comprises the following steps: dividing a computational graph running on a multi-level storage chip to obtain a plurality of search areas; searching for each search area by adopting a dynamic planning search mode to obtain an optimal segmentation fusion scheme; when it is determined that the optimal segmentation fusion scheme comprises a fusion group, the fusion group is converted into loop semantic nodes, and all operator nodes in the fusion group carry out tensor carrying on a shared storage layer of the multi-level storage chip; and modifying the computational graph according to the obtained loop semantic nodes in each search area. The optimal segmentation fusion scheme is searched for a computational graph in a dynamic planning search mode, operator nodes located in the same fusion group in the optimal fusion scheme only conduct operator calculation on a shared storage layer, and transmission of tensor data between a high-level storage layer and a low-level storage layer is avoided; and the data processing efficiency of the computational graph is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to computer technology, and in particular to a method, device, equipment and medium for compiling and optimizing a computational graph. Background Art

[0002] In artificial intelligence chips, adopting a multi-level storage design is an effective means to balance cost and performance. Its characteristics are that the lower-level storage closer to the computing unit has faster data access and smaller capacity, while the higher-level storage farther away from the computing unit has slower data access and larger capacity. The computational graph is a data structure that represents the neural network computing process. It consists of nodes and edges. Nodes represent operations, such as logical calculations or data transfer, which are ultimately performed by operators. Operators are specifically data flow operations that move tensors to be calculated from high-level storage to low-level storage, mathematical calculation operations that are performed according to specific logic, and data flow operations that move calculation results from low-level storage back to high-level storage.

[0003] However, when the computation graph is processing ultra-large-scale tensor data, it needs to frequently move tensor data between the high-level storage layer and the low-level storage layer, and increase the number of tensor data transmissions between the memory and the computing unit, as well as the data read and write operations. This will significantly reduce the data processing efficiency of the computation graph, thereby affecting the computational performance of the computation graph. Summary of the invention

[0004] An embodiment of the present invention provides a method for compiling and optimizing a computational graph to improve the data processing efficiency and overall computing performance of the computational graph.

[0005] In a first aspect, an embodiment of the present invention provides a method for compiling and optimizing a computation graph, comprising: dividing a computation graph running on a multi-level storage chip to obtain multiple search areas;

[0006] A dynamic programming search method is used to search for each of the search areas to obtain the best segmentation and fusion solution;

[0007] When it is determined that the optimal splitting and fusion scheme includes a fusion group, converting the fusion group into a loop semantic node, wherein all operator nodes in the fusion group perform tensor transport in a shared storage layer of the multi-level storage chip;

[0008] The computation graph is modified according to the loop semantic nodes obtained in each of the search areas.

[0009] In a second aspect, an embodiment of the present invention provides a computing graph compilation optimization device, including: a search area division module, used to divide the computing graph running on a multi-level storage chip to obtain multiple search areas;

[0010] The optimal segmentation and fusion solution search module is used to search for the optimal segmentation and fusion solution by using a dynamic programming search method for each search area;

[0011] A loop semantic node conversion module, used for converting the fusion group into a loop semantic node when it is determined that the optimal segmentation and fusion scheme includes a fusion group, wherein all operator nodes in the fusion group perform tensor transport in the shared storage layer of the multi-level storage chip;

[0012] A computation graph modification module is used to modify the computation graph according to the loop semantic nodes obtained in each of the search areas.

[0013] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0014] In a fourth aspect, an embodiment of the present invention provides a storage medium of computer executable instructions, on which a computer program is stored, characterized in that the program implements the method described above when executed by a processor.

[0015] The present invention searches for the best splitting and fusion scheme by adopting dynamic programming search method on the computation graph, thereby maximizing the use of historical search results to simplify the amount of calculation in the search process, and the operator nodes in the same fusion group in the best fusion scheme only perform operator calculations in the shared storage layer, thereby avoiding the transmission of tensor data between the high-level storage layer and the low-level storage layer, improving the data processing efficiency of the computation graph, and further improving the computational performance of the computation graph by converting the complex structured fusion group into a simple loop semantic node. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a method for compiling and optimizing a computational graph provided in Embodiment 1 of the present invention;

[0017] Figure 2 is a schematic diagram of the structure of a multi-level storage chip provided in Embodiment 1 of the present invention;

[0018] Figure 3 Schematic diagram of the conversion process of a cyclic semantic node provided in the first embodiment of the present invention;

[0019] Figure 4 It is a flowchart of a method for compiling and optimizing a computational graph provided in Embodiment 2 of the present invention;

[0020] Figure 5 It is a structural diagram of a computing graph compilation optimization device provided by Embodiment 3 of the present invention;

[0021] Figure 6 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0023] Embodiment 1

[0024] Figure 1 This is a flowchart of a method for compiling and optimizing a computational graph provided in the first embodiment of the present invention. This embodiment is applicable to the case of compiling and optimizing a computational graph. The method can be executed by a device for compiling and optimizing a computational graph, which can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0025] Step S101, dividing the computation graph running on the multi-level storage chip to obtain multiple search areas.

[0026] Optionally, a computation graph running on a multi-level storage chip is divided to obtain multiple search areas, including: identifying each operator node in the computation graph to obtain specified operator nodes that do not support segmentation, wherein the specified operator nodes include global normalization operator nodes and user-defined operator nodes; filtering the specified operator nodes from the computation graph to obtain a processed computation graph; dividing the processed computation graph into regions to obtain multiple search areas, wherein each search area includes multiple connected operator nodes.

[0027] Specifically, the computation graph in this embodiment includes multiple operator nodes, which represent logical computation operations such as add and mul, or data handling operations such as slice. The computation graph is represented in the form of an intermediate representation (IR). Figure 2In the multi-level storage chip shown, L3 is stored as a global storage space, which is the highest level storage layer, with the largest capacity but the slowest speed; L2 is stored as a shared storage layer, which can be accessed by multiple computing units, such as SIP, with a smaller capacity but a higher speed; L1 is stored as exclusive storage for computing units, with the smallest capacity but the fastest speed. In this embodiment, compilation optimization is mainly performed on the computational graph for processing ultra-large-scale tensors, and tensor data is the data processed by the operator node. Tensor data can have multiple dimensions, such as 64x384x768, 64x3x224x224, and 64x224x224x3, etc., and the highest dimension of the tensor represents the batch size (batch size, bs). For example, the highest dimension of each tensor data in the above example is 64. Of course, this embodiment is only an example, and does not limit the batch size of ultra-large-scale tensors processed by the computational graph.

[0028] Among them, in this embodiment, the splitting and fusion operation is mainly performed in the shared storage layer, and fusion mainly refers to IO fusion. When IO fusion is not performed, the output data of the operator nodes in the calculation graph are stored in the L3 level, which is the highest level storage layer by default. The operator loads data from L3 to L2 during calculation, and then each calculation unit SIP loads a part of the data to L1 for calculation. After the calculation is completed, it is finally written back to L3 from L1 through L2. This operator node is executed, and then the next operator node starts to load data from L3. This method will obviously increase the number of data transmissions at different storage levels. When IO fusion is used, the output of the previous operator node will not be written back to L3, but temporarily stored in L2, so the next operator node can directly load data from L2, that is, only the previous operator node loads data from L3, and the next operator node writes the calculation results after the two operator logics are continuously executed back to L3. At this time, it is considered that the two operator nodes are fused into one operator node. However, this requires that the output of the previous operator can be completely temporarily stored on L2. If the tensor is too large to be temporarily stored on L2, it needs to be split.

[0029] It should be noted that, in this embodiment, before performing splitting and fusion, it is necessary to first identify each operator node in the calculation graph to obtain and filter the specified operator nodes that do not support splitting, such as global normalization operator nodes and user-defined operator nodes, etc. Of course, this embodiment is only an example and does not limit the specific type of the specified operator node. And the operator nodes to be fused need to have a connection relationship, so in this embodiment, the filtered calculation graph will be divided into regions to obtain multiple search regions, and each search region includes multiple connected operator nodes. By dividing the calculation graph into regions, invalid fusion judgments on unconnected operator nodes can be avoided.

[0030] Step S102: A dynamic programming search method is used to search for each search area to obtain the best segmentation and fusion solution.

[0031] Optionally, a dynamic programming search method is used to search for each search area to obtain the best splitting and fusion scheme, including: using a dynamic programming search method to search for each search area to obtain all splitting and fusion schemes for operator nodes; calculating the splitting benefit of each splitting and fusion scheme in each search area, and taking the splitting and fusion scheme with the largest splitting benefit as the best splitting and fusion scheme in each search area.

[0032] Specifically, in this embodiment, when searching for the best splitting and fusion scheme for each search area, a dynamic programming search method is used to search and obtain all splitting and fusion schemes for the operator nodes, and the splitting and fusion scheme with the largest splitting benefit is used as the best splitting and fusion scheme in each search area. Splitting benefits will only exist if there is a fusion group in the splitting and fusion scheme, and there will be no splitting benefits for the splitting scheme without a fusion group, because for each operator node in the same fusion group, the input of the previous operator node is not written back to L3, but temporarily stored in L2 and used as input for the subsequent operator node until the last operator node is written back to L3. Then these operator nodes can be called a fusion group, and the same splitting method is used inside the fusion group, so that the output of the previous operator node can be used as the input of the next operator node. Since the computation graph is represented in the form of IR, the representation method needs to be modified on the IR. Fusion is used to wrap the IRs corresponding to multiple operator nodes that can be fused. For example, two IRs are packaged into a fusion group IR, which contains two operators. In form, No.1 add -> No.2 mul becomes No.1 fusion (add+mul), and the two IRs become one IR, that is, two separate operator nodes become a fusion group. Of course, this implementation only takes the fusion of two operator nodes as an example. The fusion method of multiple operator nodes is roughly the same, and will not be repeated in this implementation.

[0033] The purpose of the splitting in this implementation is to cache the output tensor of the previous operator after splitting on L2 as input to the next operator node, in order to achieve IO fusion. Splitting and fusion are closely related. The dynamic search method is to search which operator nodes are fused into a fusion group, and which dimensions of the processed tensor data need to be split so that it can be cached by L2, so that the calculation graph can reduce the data handling time the most, that is, the maximum benefit. After splitting, the shape will become smaller. For example, 6x3x224x224 can be split into 3 2x3x224x224. This also means that the operator node needs to be processed three times, that is, the operator node changes from one to three to process these data. The tensor data processed each time becomes bs2. Therefore, when splitting, bs2 is split from the original L3 bs6 tensor data each time, and moved to L2 to call the operator calculation. After the three splits and calculations are completed, the results of the bs2 on the three L2s need to be moved back to L3 and spliced ​​into L3 bs6 for the next operator node as input.

[0034] In a specific implementation, in the dynamic programming scheme search in this embodiment, when calculating the benefits during the search process, it is assumed that some operator nodes have been calculated after L2 IO fusion. The basis for IO fusion is that the tensor is divided into a suitable size that does not exceed the L2 capacity. That is, first fuse some operator nodes into a whole fusion group, then continuously adjust the size of the tensor split, and record the benefits. The benefit of a single operator node is the basis of the search. The split benefit of the split fusion scheme is calculated based on the fusion group. The scheme search process is to find all possible fusion groups and score them to find the process with the highest score. Assuming that a search area includes n operator nodes, when searching using dynamic programming search, the specific process is as follows:

[0035] Step 1: Establish the profit calculation method when the operator node x to the operator node y is used as a fusion group as a basic function. For the fusion group from the operator node x to the operator node y, first calculate the peak storage occupied by the original tensor size and the quantitative relationship with the L2 storage capacity, and then divide it to at least this number. According to this number of divisions, update the size of the tensor in the entire fusion group, recalculate the peak storage usage, and if it is higher than the L2 storage capacity, continue to increase the number of divisions and reduce the tensor size until the tensor can be cached by L2 after division. Calculate the fusion group profit at this time and record it. Continue to calculate and record the profits under other divisions according to the preset optimization search method, such as aligning the divisions to the number of SIPs for better hardware performance, so as to select the best division method and record it as B(x, y). Step 2: Let x=1, y=1, that is, when there is only one operator node, then B(1, 1) will be recorded as the best solution O(1, 1) from operator node 1 to operator node 1. Step 3: Let x=1, y=2. Then O(1, 1) is known. Calculate B(1, 2) and B(2, 2) according to step 1 above. Compare O(1,1) + B(2,2) and B(1,2), which one is greater. The solution with the maximum benefit is the best solution for operator node 1 to operator node 2, recorded as O(1,2). Step 4: Similarly, according to the state transfer equation: O(1,n) = Max{ B(1,n), O(1,1)+B(2,n), O(1,2)+B(3,n), O(1,3)+B(4,n), ...., O(1,k)+B(k+1,n), ...., O(1,n-3)+B(n-2,n), O(1,n-2)+B(n-1,n), O(1,n-1)+B(n,n)}, continue to calculate O(1,3), O(1,4). . . . Until O(1,n), the best splitting and fusion solution for the current search area containing n operator nodes can be obtained. For example, when the search area includes five operator nodes ABCDE, in the process of dynamic programming search, the segmentation and fusion methods such as A+B+C+D+E, AB+C+D+E, ABC+D+E, A+BC+D+E, ABCD+E, AB+CD+E, A+BCD+E, A+B+C+DE, AB+C+DE, ABC+DE, A+BC+DE, AB+CDE, A+BCDE, ABCDE will be searched, and each fusion group will score the segmentation time and segmentation benefits according to the L2 capacity size to select the best segmentation and fusion plan in the search area. In addition, since the dynamic programming search method can make full use of the searched plans without calculating all the search plans, the search efficiency is improved.For example, when searching for the best splitting and fusion scheme from A to D, assuming that the benefit of ABCD fusion is greater than the benefit of A+B+C+D without fusion, then when continuing to search from A to E, the previous search results can be fully utilized, that is, when searching for [ABCD]+E and [A+B+C+D]+E, it is obviously not necessary to calculate them, because the first part of the benefit in the former scheme is the same as ABCD, and the first part of the benefit in the latter scheme is the same as [A+B+C+D], and the latter is definitely smaller than the former. Therefore, in this implementation, search optimization will be performed based on the existing information in the search process, and only necessary searches will be performed, but it can still ensure that the global best splitting and fusion scheme is searched.

[0036] Step S103: when it is determined that the best segmentation and fusion scheme includes a fusion group, the fusion group is converted into a cyclic semantic node.

[0037] Optionally, converting the fusion group into a loop semantic node includes: determining sub-operator nodes obtained by splitting each operator node in the fusion group according to the number of splits; inserting slice nodes and anti-slice nodes representing tensor transfer between the highest-level storage layer and the shared storage layer in the fusion group, wherein the number of anti-slice nodes is the same as the number of splits; combining the sub-operator nodes, slice nodes and anti-slice nodes to construct a sub-fusion group, wherein the sub-fusion group includes sub-operator nodes of different types, and the number of sub-fusion groups is the same as the number of splits; connecting each sub-fusion group with a connection node to obtain an expanded fusion group, and converting the expanded fusion group into a loop semantic node, wherein the sub-fusion groups are used to process sub-tensor data of the same size, and the sum of each sub-tensor data is the original processed tensor data of the fusion group.

[0038] Optionally, converting the expanded fusion group into a loop semantic node includes: performing computational logic identification on all slices in the expanded fusion group to obtain slices with the same computational logic, wherein the offset information of each slice with the same computational logic is different; merging the slices with the same computational logic into a loop slice with loop semantics, wherein the number of loops of the loop slice is the same as the number of divisions; and converting the expanded fusion group into a loop semantic node based on the loop slice.

[0039] Specifically, in this embodiment, when it is determined that the best splitting and fusion scheme includes a fusion group, since the operator nodes in the fusion group will be processed according to the full expansion when splitting the tensor data, it will cause one operator node on the calculation graph to become multiple, thereby increasing the complexity of the calculation graph, so that the compilation processing time of the entire network will be longer. In this embodiment, after converting the fusion group into a loop semantic node, since the number of loops can be marked in the loop semantic node, it is only necessary to express the operator nodes involved in the processing flow of a single tensor data in the structural expression, and the loop semantic node will be automatically scheduled multiple times during the operation process, so as to achieve the same effect as the global expansion, and the loop semantic node can be called a while node.

[0040] In a specific implementation, Figure 3 The figure shows a schematic diagram of the conversion process of the loop semantic node. When it is determined that a fusion group includes three operator nodes A, B and C, and the highest dimension of the original tensor data processed by the fusion group is bs6, when the number of splits for the fusion group is determined to be 3 according to the optimal splitting and fusion scheme, each operator node can be split into three sub-operator nodes, and the tensor data processed by each sub-operator node is reduced to bs2. For example, the operator node A is split into three sub-operator nodes a, the operator node is split into three sub-operator nodes b, and the operator node C is split into three sub-operator nodes c, and the functions of the three sub-operator nodes split from each operator node are the same. Of course, this embodiment is only an example, and the number of sub-operator nodes split from each operator node is not limited. In this implementation, a slice node slice for transferring tensor data from L3 to L2 is inserted at the beginning of the fusion group, and a deslice node deslice for transferring tensor data from L2 to L3 is added at the end of the fusion group. The number of slice nodes and deslice nodes is 3, which is the same as the number of divisions of each operator node in the fusion group. Figure 3Only a linear structure fusion group with a single input and a single output is used as an example for explanation. In actual applications, the fusion group can also be a tree structure with multiple inputs and a single output. In this case, the number of anti-slice nodes is still the same as the number of splits, but the number of slice nodes is related to the input operator nodes and the number of splits. Therefore, the number of slice nodes and anti-slice nodes is no longer the same. Of course, this embodiment is only an example, and the specific number of slice nodes is not limited. In this embodiment, a sub-operator node is extracted from the sub-operator nodes split from each operator node and connected in the execution order. Then, the slice node is connected to the first operator node, and the anti-slice node is connected to the last operator node to construct a sub-fusion group, for example, slice-abc-deslice. Therefore, each sub-fusion group includes sub-operator nodes of different types. Since each operator node can be split into three identical sub-operator nodes, three sub-fusion groups can be obtained by the above construction method, and each obtained sub-fusion is connected using a connection node, such as concat, to obtain an expanded fusion group. Figure 3 It can be seen that the expanded fusion group splits the operator nodes due to tensor segmentation, and the number of nodes increases significantly. In this implementation, each sub-fusion group processes the bs2 tensor data separately, and when the operator nodes in the same sub-fusion group perform operations, the output of the previous sub-operator node is no longer written back to L3, but temporarily stored on L2 to be used as input for the following operator nodes until the last sub-operator node is written back to L3. For example, after the a operator node loads data from L3 and processes it to obtain the first processing result, the first processing result is not written back to L3 but directly transmitted to the b operator node and used as the input of the b operator node. Then the b operator node processes according to the first processing result to obtain the second processing result. As a result, the second processing result is not written back to L3 but directly transmitted to the c operator node and used as the input of the c operator node. Finally, the c operator node processes the second processing result to obtain the third processing result, and finally writes the third processing result back to L3. Since it can be seen that the calculation process of the sub-operator nodes in the same sub-fusion group is all executed on L2, only the data handling operation with L3 is involved when the initial data is obtained and the final processing result is written back. Therefore, compared with the operation method of processing once and writing back once in the prior art, the number of data transmissions is significantly reduced. Since each sub-fusion group has the same structure, processing logic, and processing tensor data size, the expanded fusion group can be converted into a loop semantic node.

[0041] Among them, the expanded fusion group is converted into a loop semantic node, that is, multiple sub-fusion groups with the same structure, processing logic and processing tensor data size in the fusion group are merged into a while loop semantic node, and in the specific operation, the slice nodes with the same calculation logic in the expanded fusion group are converted into loop slices with loop semantics. For example, three slice slice nodes are converted into a loopslice loop slice, and three deslice anti-slices are converted into a loopdeslice loop anti-slice. The number of loopslice and loopdeslice is 3, and the corresponding three sub-fusion groups are merged into one, and loopslice and loopdeslice are used to connect to obtain the while loop semantic node. The input and output of the while loop semantic node are complete bs6 tensor data, and the tensor data processed in each loop are all bs2, so the number of operator nodes in the expanded fusion group is reduced through the loop processing method. In this implementation, the concat processing of the L3 level in the expanded fusion group is optimized to avoid multiple L3 concatenation and transfer. The L3 output of the While loop semantic node is only allocated a storage space of the original tensor size. After each slicing loop is executed, the tensor data of each small-scale calculation result on L2 is directly transferred from L2 to the corresponding position of L3output by using different offsets, thereby avoiding the transfer of tensors on L3.

[0042] Step S104, modifying the computation graph according to the loop semantic nodes obtained in each search area.

[0043] Optionally, the computation graph is modified according to the loop semantic nodes obtained in each search area, including: determining a designated position of each fusion group on the computation graph; and replacing the fusion group with a matching loop semantic node at the designated position.

[0044] Specifically, in this implementation, after obtaining the fusion groups in all search areas of the computation graph and converting the fusion groups into loop semantic nodes, the designated positions of each fusion group in the computation graph are determined, and the loop semantic nodes matched by the fusion groups are replaced with the operator nodes at that position, thereby optimizing the computation graph. The optimized computation graph has a simpler structure and more efficient internal processing, which avoids additional IO handling and significantly improves the overall computing performance of the computation graph.

[0045] This implementation method searches for the best splitting and fusion solution by using dynamic programming search on the computation graph, thereby maximizing the use of historical search results to simplify the amount of computation in the search process, and the operator nodes in the same fusion group in the best fusion solution only perform operator calculations in the shared storage layer, thereby avoiding the transmission of tensor data between the high-level storage layer and the low-level storage layer, improving the data processing efficiency of the computation graph, and further improving the computational performance of the computation graph by converting the complex structured fusion group into a simple loop semantic node.

[0046] Embodiment 2

[0047] Figure 4 This is a flowchart of a method for compiling and optimizing a computational graph provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and specifically describes how to calculate the segmentation benefit of each segmentation and fusion scheme in each search area during the dynamic programming search process in the above embodiment. Figure 4 As shown, the method includes:

[0048] Step S201, determine whether the segmentation scheme includes a fusion group, if so, execute step S202, otherwise, execute step S204.

[0049] Specifically, the splitting benefit of the splitting and fusion scheme in this embodiment is calculated based on the fusion group. If the operator node in the calculation is not fused into any fusion group, there is no benefit. The tensor data of all operator nodes in each fusion group adopts the same splitting method, so that the size of the splits matches, and the output of the previous operator can be directly given to the next operator as input. Therefore, before calculating the splitting benefit of the splitting and fusion scheme in this embodiment, it is first necessary to determine whether the fusion group is included in the splitting and fusion scheme.

[0050] Step S202, calculating the segmentation benefit of each fusion group.

[0051] Optionally, the sharding benefit of each fusion group is calculated, including: obtaining the peak storage usage of all operator nodes in each fusion group in the shared storage layer, and determining the number of sharding times for the fusion group based on the peak storage usage and the storage capacity of the shared storage layer; obtaining the operator overhead of each operator node in the fusion group in the highest-level storage layer before sharding, and the operator overhead in the shared storage layer after sharding according to the number of sharding times, wherein the shared storage layer is located below the highest-level storage layer; determining the sharding benefit of each operator node in the fusion group in the shared storage layer based on the number of sharding times, the operator overhead of the highest-level storage layer, and the operator overhead of the shared storage layer; and obtaining the sharding benefit of the fusion group based on the sharding benefit of each operator node in the shared storage layer.

[0052] Optionally, the slicing benefit of the fusion group is obtained based on the slicing benefit of each operator node in the shared storage layer, including: obtaining the transportation overhead between the highest-level storage layer and the shared storage layer of the input operator nodes and the output operator nodes in the fusion group; obtaining the sum of the slicing benefits of all operator nodes in the fusion group in the shared storage layer, and taking the difference between the sum of the slicing benefits and the transportation overhead as the slicing benefit of the fusion group.

[0053] In a specific implementation, when the computation graph includes five operator nodes ABCDE, the original processing tensor data is bs6, when a splitting and fusion scheme is obtained during the search: AB is fused, the tensor is split to bs2; C remains as a single operator at L3 and is not split; DE is fused, and the tensor is split to bs3. The benefit of a single operator only considers the normal overhead of the operator's own tensor in L3, and the total overhead when the tensor is in L2 when it is split into multiple processes. The benefits of a single operator in various splits, which dimensions to split, how many parts to split into, and the size of each part are the basic information for dynamic programming search, and the operator nodes in the fusion group that are not fused have no splitting benefits. Therefore, in this implementation, only the splitting benefits of each operator node in the fusion group are considered, and the splitting benefits of the entire fusion group are obtained based on the splitting benefits of each operator node in the fusion group. Among them, when calculating the splitting benefit of fusion group AB, the splitting benefits of operator nodes A and B can be obtained respectively. In this implementation, the splitting benefit of operator node A is taken as an example for explanation. When obtaining the splitting benefit of operator node A, the storage usage peak value of all operator nodes in fusion group AB in L2 is first obtained, and the number of splits for fusion group AB is determined based on the storage usage peak value and the storage capacity of L2. For example, the split is first performed according to the highest dimension. If the highest dimension still exceeds the storage size of L2 after splitting, the next higher dimension is continued to be split to ensure continuity. Then the following formula (1) is used to calculate the splitting benefit of operator node A on L2 after splitting:

[0054] The sharding benefit of a single operator node = L3 operator overhead - number of sharding times * L2 operator overhead (1)

[0055] Among them, the L3 operator overhead is the overhead when the tensor data of a single operator node is stored in L3 without being split, and the L2 operator overhead is the overhead when a single operator node is split once when it exists in L2. Since the actual execution strategy of the operator node at different storage levels and different tensor shapes is generated according to a known specific algorithm, including the size of the data transferred each time between high- and low-level storage, the shape of the tensor processed each time by the computing unit, etc. The frequency of the AI ​​hardware platform, the data processing capabilities of the computing units under different operators, the data transfer rate, the task issuance overhead, etc. are also known. Therefore, based on the above information, the following formula (2) can be used to obtain the overhead of a single operator node at different storage levels:

[0056] Operator cost = fixed cost of calling the operator + first input tensor size / data transfer rate + max (single input tensor size / computing unit data processing capacity, single input tensor size / data transfer rate + single output tensor size / data transfer rate) + final result tensor size / data transfer rate (2)

[0057] Among them, in this embodiment, after obtaining the slicing benefits of each single operator node in the fusion group AB, the slicing benefits of the fusion group AB can be obtained based on the slicing benefits of each operator node in the shared storage layer. When performing the slicing benefits of the fusion group, since the fusion group composed of multiple operator nodes only needs to be called once during execution, multiple fixed operator call overheads are eliminated to obtain basic benefits. Therefore, when calculating the benefits of a certain slicing of the fusion group, in addition to accumulating the slicing benefits of all internal operators, it is necessary to deduct the transportation cost of the tensor from L2 to L3 each time the last output operator node splits, and the transportation cost from L3 to L2 after each slicing of each input operator node, and merge the transportation costs in the above two directions into the transportation cost between L3 and L2. In addition, the sum of the slicing benefits of all operator nodes in the fusion group in L2 is obtained, and the slicing benefit of the fusion group is obtained based on the following formula (3):

[0058] Fusion group splitting benefit = the sum of the splitting benefits of all operator nodes - the transport overhead between L3 and L2 (3)

[0059] It is worth mentioning that in this implementation, when the operator node is split into multiple executions, the execution strategy of the operator will change, and the hardware characteristics have problems with the efficiency of handling tensors of different shapes, resulting in the calculation of the computing unit and the overhead of tensor handling may become more than if no splitting is done, and there may be multiple repeated weight handling and other factors. Splitting a single operator node does not necessarily obtain positive benefits. However, when multiple operator nodes are fused continuously, even if the split benefit of a single operator node in the middle is negative, it may maximize the benefit of the entire fusion group. For example, if the split benefits of 5 consecutive operator nodes are 2, 2, 2, -1, 2, then it is obvious that fusing the fourth operator node with a negative benefit together with the fifth operator node will obtain more benefits, rather than fusing only the first three operator nodes. The actual situation will be more complicated, and in this implementation, the dynamic programming search technology introduced above can be used to search for the global optimal splitting solution to obtain the best splitting and fusion solution.

[0060] Step S203, obtaining the total segmentation benefits of all fusion groups in each segmentation scheme, and using the total segmentation benefits as the segmentation benefits of each segmentation and fusion scheme.

[0061] Specifically, for the splitting and fusion scheme: AB fusion, the tensor is split to bs2; C remains as an L3 single operator and is not split; DE fusion, when the tensor is split to bs3, after obtaining the sum of the splitting benefits of the fusion group AB and the splitting benefits of the fusion group DE based on the above method, the splitting benefits of the fusion group AB and the fusion group DE can be added to obtain the total splitting benefit, and the total splitting benefit is used as the splitting benefit of the splitting scheme. Of course, this embodiment only takes the splitting benefit of obtaining a splitting and fusion scheme as an example for explanation. The process of obtaining the splitting benefits of other splitting and fusion schemes is roughly the same, and will not be repeated in this embodiment.

[0062] Step S204, directly determine that the splitting benefit of the splitting and merging solution is 0.

[0063] This implementation method searches for the best splitting and fusion solution by using dynamic programming search on the computation graph, thereby maximizing the use of historical search results to simplify the amount of computation in the search process, and the operator nodes in the same fusion group in the best fusion solution only perform operator calculations in the shared storage layer, thereby avoiding the transmission of tensor data between the high-level storage layer and the low-level storage layer, improving the data processing efficiency of the computation graph, and further improving the computational performance of the computation graph by converting the complex structured fusion group into a simple loop semantic node.

[0064] Embodiment 3

[0065] Figure 5 This is a schematic diagram of the structure of a computing graph compilation optimization device provided by Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a search area division module 310, an optimal segmentation and fusion solution search module 320, a loop semantic node conversion module 330 and a calculation graph modification module 340.

[0066] The search area division module 310 is used to divide the computation graph running on the multi-level storage chip to obtain multiple search areas;

[0067] The optimal segmentation and fusion solution search module 320 is used to search for the optimal segmentation and fusion solution in each search area by using a dynamic programming search method;

[0068] A loop semantic node conversion module 330 is used to convert the fusion group into a loop semantic node when it is determined that the best splitting and fusion scheme includes a fusion group, wherein all operator nodes in the fusion group perform tensor transfer in a shared storage layer of a multi-level storage chip;

[0069] The computation graph modification module 340 is used to modify the computation graph according to the loop semantic nodes obtained in each search area.

[0070] Optionally, a search region partitioning module is used to identify each operator node in the computation graph to obtain a designated operator node that does not support segmentation, wherein the designated operator node includes a global normalization operator node and a user-defined operator node;

[0071] Filter the specified operator nodes from the computation graph to obtain the processed computation graph;

[0072] The processed computation graph is divided into regions to obtain multiple search regions, wherein each search region includes multiple connected operator nodes.

[0073] Optional, optimal segmentation and fusion solution search module includes:

[0074] A splitting and fusion scheme search unit is used to search for each search area using a dynamic programming search method to obtain all splitting and fusion schemes for operator nodes;

[0075] The optimal segmentation and fusion scheme determination unit is used to calculate the segmentation benefit of each segmentation and fusion scheme in each search area, and take the segmentation and fusion scheme with the largest segmentation benefit as the optimal segmentation and fusion scheme in each search area.

[0076] Optionally, the optimal splitting and fusion scheme determining unit includes a splitting and fusion scheme splitting and fusion scheme splitting and fusion scheme splitting and fusion scheme calculating unit, which is used to directly determine that the splitting and fusion scheme splitting and fusion scheme splitting and fusion scheme is 0 when it is determined that the splitting and fusion scheme does not include a fusion group;

[0077] When it is determined that the segmentation and fusion scheme includes a fusion group, the segmentation benefit of each fusion group is calculated, the total segmentation benefit of all fusion groups in each segmentation scheme is obtained, and the total segmentation benefit is used as the segmentation benefit of each segmentation and fusion scheme.

[0078] Optionally, a splitting benefit calculation unit of the splitting and fusion scheme is used to obtain the storage usage peak value of all operator nodes in each fusion group in the shared storage layer, and determine the number of splits for the fusion group according to the storage usage peak value and the storage capacity of the shared storage layer;

[0079] Get the operator overhead of each operator node in the fusion group before splitting in the tensor storage in the highest level storage layer, and the operator overhead in the shared storage layer after splitting according to the number of splits, where the shared storage layer is located below the highest level storage layer;

[0080] Determine the sharding benefit of each operator node in the fusion group in the shared storage layer according to the number of sharding times, the operator cost of the highest-level storage layer, and the operator cost of the shared storage layer;

[0081] The slicing benefits of the fusion group are obtained based on the slicing benefits of each operator node in the shared storage layer.

[0082] Optionally, the splitting benefit calculation unit of the splitting and fusion scheme is further used to obtain the transport overhead of the input operator nodes and the output operator nodes in the fusion group between the highest-level storage layer and the shared storage layer;

[0083] Get the sum of the split benefits of all operator nodes in the fusion group in the shared storage layer, and use the difference between the sum of the split benefits and the transportation overhead as the split benefit of the fusion group.

[0084] Optionally, a cyclic semantic node conversion module is used to determine the sub-operator nodes obtained by segmenting each operator node in the fusion group according to the number of segmentations;

[0085] Inserting slice nodes and anti-slice nodes representing tensor transfer between the highest-level storage layer and the shared storage layer into the fusion group, wherein the number of anti-slice nodes is the same as the number of slices;

[0086] The sub-operator nodes, the slice nodes and the anti-slice nodes are combined to form a sub-fusion group, wherein the sub-fusion group includes sub-operator nodes of different types, and the number of sub-fusion groups is the same as the number of segmentation times;

[0087] Each sub-fusion group is connected by a connection node to obtain an expanded fusion group, and the expanded fusion group is converted into a loop semantic node, wherein the sub-fusion groups are respectively used to process sub-tensor data of the same size, and the sum of each sub-tensor data is the original processed tensor data of the fusion group.

[0088] Optionally, the cyclic semantic node conversion module is further used to perform computational logic identification on all slices in the expanded fusion group to obtain slices with the same computational logic, wherein the offset information of each slice with the same computational logic is different;

[0089] Merge slices with the same computational logic into a loop slice with loop semantics, where the number of loops in the loop slice is the same as the number of slices;

[0090] The unrolled fusion groups are converted into loop semantic nodes based on loop slices.

[0091] Optionally, a computational graph modification module is used to determine a specified position of each fusion group on the computational graph; and replace the fusion group with a matching loop semantic node at the specified position.

[0092] The computation graph compilation optimization device provided in the embodiment of the present invention can execute the computation graph compilation optimization method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0093] Embodiment 4

[0094] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention is shown in FIG. Figure 6 As shown, the computer device includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of the processor 610 in the computer device can be one or more. Figure 6 A processor 610 is taken as an example; the processor 610, memory 620, input device 630 and output device 640 in the computer device can be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0095] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for compiling and optimizing the computational graph in the embodiment of the present invention. The processor 610 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 620, that is, implements the above-mentioned method for compiling and optimizing the computational graph.

[0096] The computation graph running on the multi-level storage chip is divided to obtain multiple search areas;

[0097] A dynamic programming search method is used to search for each search area to obtain the best segmentation and fusion solution;

[0098] When it is determined that the best splitting and fusion scheme includes a fusion group, the fusion group is converted into a loop semantic node, wherein each operator node in the fusion group performs operator operation in a shared storage layer of a multi-level storage chip;

[0099] The computational graph is modified according to the loop semantic nodes obtained in each search area.

[0100] The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 620 may further include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0101] The input device 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 640 may include a display device such as a display screen.

[0102] Embodiment 5

[0103] Embodiment 5 of the present invention further provides a storage medium including computer executable instructions, which are used to execute a method for compiling and optimizing a computational graph when executed by a computer processor;

[0104] The computation graph running on the multi-level storage chip is divided to obtain multiple search areas;

[0105] A dynamic programming search method is used to search for each search area to obtain the best segmentation and fusion solution;

[0106] When it is determined that the best splitting and fusion scheme includes a fusion group, the fusion group is converted into a loop semantic node, wherein each operator node in the fusion group performs operator operation in a shared storage layer of a multi-level storage chip;

[0107] The computational graph is modified according to the loop semantic nodes obtained in each search area.

[0108] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention, whose computer executable instructions are not limited to the above method operations, can also execute related operations in the compilation optimization method of the computational graph provided by any embodiment of the present invention.

[0109] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0110] It is worth noting that in the above-mentioned embodiment of the compilation optimization device according to the computational graph, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0111] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for compiling and optimizing a computational graph, characterized in that: include: The computation graph running on the multi-level storage chip is divided to obtain multiple search areas; A dynamic programming search method is used to search for each of the search areas to obtain the best segmentation and fusion solution; When it is determined that the optimal splitting and fusion scheme includes a fusion group, converting the fusion group into a loop semantic node, wherein all operator nodes in the fusion group perform tensor transport in a shared storage layer of the multi-level storage chip; The computation graph is modified according to the loop semantic nodes obtained in each of the search areas.

2. The method according to claim 1, characterized in that The step of dividing the computation graph running on the multi-level storage chip to obtain multiple search areas includes: Identify each operator node in the computation graph to obtain a designated operator node that does not support segmentation, wherein the designated operator node includes a global normalization operator node and a user-defined operator node; Filtering the designated operator node from the computation graph to obtain a processed computation graph; The processed computation graph is divided into regions to obtain a plurality of search regions, wherein each of the search regions includes a plurality of connected operator nodes.

3. The method according to claim 1, characterized in that The method of using a dynamic programming search method to search for each search area to obtain the best segmentation and fusion solution includes: A dynamic programming search method is used to search for each of the search areas to obtain all the segmentation and fusion solutions for the operator nodes; The segmentation benefit of each segmentation and fusion scheme in each search area is calculated, and the segmentation and fusion scheme with the largest segmentation benefit is used as the optimal segmentation and fusion scheme in each search area.

4. The method according to claim 3, characterized in that The calculation of each of the search areas The sharding benefits of each sharding and fusion scheme include: When it is determined that the splitting and fusion scheme does not include a fusion group, the splitting benefit of the splitting and fusion scheme is directly determined to be 0; When it is determined that the splitting and fusion scheme includes a fusion group, the splitting benefits of each fusion group are calculated, the total splitting benefits of all fusion groups in each splitting scheme are obtained, and the total splitting benefits are used as the splitting benefits of each splitting and fusion scheme.

5. The method according to claim 4, characterized in that The calculating the splitting benefits of each fusion group includes: Obtaining the storage usage peak of all operator nodes in each of the fusion groups in the shared storage layer, and determining the number of splits for the fusion group according to the storage usage peak and the storage capacity of the shared storage layer; Obtaining the operator overhead of the tensor stored in the highest-level storage layer before each operator node in the fusion group is split, and the operator overhead in the shared storage layer after being split according to the number of splits, wherein the shared storage layer is located at the lower layer of the highest-level storage layer; Determine the slicing benefit of each operator node in the fusion group in the shared storage layer according to the number of slicing times, the operator overhead of the highest-level storage layer, and the operator overhead of the shared storage layer; The splitting benefit of the fusion group is obtained according to the splitting benefit of each operator node in the shared storage layer.

6. The method according to claim 5, characterized in that The obtaining the slicing benefit of the fusion group according to the slicing benefit of each operator node in the shared storage layer includes: Obtaining the transport overhead of the input operator nodes and the output operator nodes in the fusion group between the highest-level storage layer and the shared storage layer; Obtain the sum of the split benefits of all operator nodes in the fusion group in the shared storage layer, and convert the The difference between the splitting benefit and the transport overhead is used as the splitting benefit of the fusion group.

7. The method according to claim 5, characterized in that The converting the fusion group into a cyclic semantic node comprises: Determine the sub-operator nodes obtained by segmenting each operator node in the fusion group according to the number of segmentations; Inserting a slice node and an anti-slice node representing tensor transfer between the highest-level storage layer and the shared storage layer into the fusion group, wherein the number of the anti-slice nodes is the same as the number of slicing times; Combining the sub-operator nodes, the slice nodes and the anti-slice nodes to construct a sub-fusion group, wherein the sub-fusion group includes the sub-operator nodes of different types, and the number of the sub-fusion groups is the same as the number of segmentation times; Each of the sub-fusion groups is connected by a connection node to obtain an expanded fusion group, and the expanded fusion group is converted into the loop semantic node, wherein the sub-fusion groups are respectively used to process sub-tensor data of the same size, and the sum of each of the sub-tensor data is the original processed tensor data of the fusion group.

8. The method according to claim 7, characterized in that The converting the expanded fusion group into the loop semantic node comprises: Performing calculation logic identification on all slices in the expanded fusion group to obtain slices with the same calculation logic, wherein the offset information of the slices with the same calculation logic is different; Merge the slices with the same computational logic into a loop slice with loop semantics, wherein the number of loops of the loop slice is the same as the number of slices; The expanded fusion group is converted into the loop semantic node based on the loop slice.

9. The method according to claim 1, characterized in that: The modifying the computation graph according to the loop semantic nodes obtained in each of the search areas includes: Determine a designated position of each of the fusion groups on the computational graph; The fusion group is replaced at the specified position with the matching loop semantic node.

10. A computing graph compilation optimization device, characterized in that: include: A search area partitioning module is used to partition the computational graph running on the multi-level storage chip to obtain multiple search areas; The optimal segmentation and fusion solution search module is used to search for the optimal segmentation and fusion solution by using a dynamic programming search method for each search area; A loop semantic node conversion module, used for converting the fusion group into a loop semantic node when it is determined that the optimal segmentation and fusion scheme includes a fusion group, wherein all operator nodes in the fusion group perform tensor transport in the shared storage layer of the multi-level storage chip; A computation graph modification module is used to modify the computation graph according to the loop semantic nodes obtained in each of the search areas.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

12. A computer executable instruction storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images