Compiling optimization method and device of computational graph based on multi-level storage chip

Abstract

The invention discloses a compilation optimization method and device of a computational graph based on a multi-level storage chip. The method comprises the following steps: screening calculation graphs running on a multi-level storage chip to obtain candidate operator nodes; obtaining the dynamic type of each current candidate operator node; performing forward search and backward search on candidate operator nodes in the computational graph according to the dynamic types to obtain fusion operator nodes, updating the computational graph according to the fusion operator nodes to obtain an optimized computational graph, and screening out the candidate operator nodes meeting fusion conditions from the computational graph and the dynamic types of the candidate operator nodes to obtain a fusion result. According to the method, the candidate operator nodes are selected, the candidate operator nodes are searched according to the dynamic types to obtain the fusion operator nodes, and the candidate operator nodes located in the same fusion operator node only carry out tensor carrying in the shared storage layer, so that the carrying frequency of tensor data among different storage layers is avoided, and the data processing efficiency and the calculation performance of the computational graph are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to computer technology, and in particular to a compilation optimization method and device for a computational graph based on a multi-level storage chip. Background Art

[0002] The farther the memory is from the computing unit on a multi-level storage chip, the larger its memory capacity is and the lower the data access efficiency is. A computational graph is a data structure that represents the computational process of a neural network. The computational process consists of operators one by one. There are data dependencies between operators, and data transfer will occur on the outermost storage of the multi-level storage chip.

[0003] When the computation graph is processing data, the computation process of a single operator reads the data from the outermost storage and moves it to the storage closest to the computation unit, then loads it into the register for computation, writes it back to the innermost storage after the computation is complete, and finally moves it to the outermost storage. On the inference side, the dependent data between operators only needs to reside in the storage temporarily, and the final computation result does not use this part of the data, that is, the intermediate result data. When the intermediate result data is transferred through the outer-level storage, the access rate of its write-back and read-out is low, which significantly reduces the data processing efficiency of the computation graph, thereby affecting the computation performance of the computation graph. Summary of the invention

[0004] An embodiment of the present invention provides a compilation optimization method for a computational graph based on a multi-level storage chip to improve the data processing efficiency and overall computing performance of the computational graph.

[0005] In a first aspect, an embodiment of the present invention provides a method for compiling and optimizing a computation graph, comprising: screening a computation graph running on a multi-level storage chip to obtain candidate operator nodes;

[0006] Obtaining the dynamic type of each of the current candidate operator nodes, wherein the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output, and multiple-input multiple-output;

[0007] According to the dynamic type, forward search and backward search are performed on the candidate operator nodes in the calculation graph to obtain a fusion operator node, and according to the fusion operator node, the calculation graph is updated to obtain an optimized calculation graph,

[0008] Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

[0009] In a second aspect, an embodiment of the present invention provides a computing graph compilation optimization device, including: a candidate operator node screening module, used to screen the computing graph running on the multi-level storage chip to obtain candidate operator nodes;

[0010] A dynamic type acquisition module, used to acquire the dynamic type of each of the current candidate operator nodes, wherein the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output and multiple-input multiple-output;

[0011] An optimized calculation graph acquisition module is used to perform forward search and backward search on the candidate operator nodes in the calculation graph according to the dynamic type to obtain a fusion operator node, and update the calculation graph according to the fusion operator node to obtain an optimized calculation graph,

[0012] Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

[0013] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0014] In a fourth aspect, an embodiment of the present invention provides a storage medium of computer executable instructions, on which a computer program is stored, characterized in that the program implements the method described above when executed by a processor.

[0015] The present invention screens out candidate operator nodes that meet the fusion conditions and the dynamic type of each candidate operator node from the calculation graph, and searches the candidate operator nodes according to the dynamic type to obtain the fusion operator node. Since the candidate operator nodes located in the same fusion operator node only perform tensor transfer in the shared storage layer, the number of times tensor data is transferred between different storage layers is avoided, thereby improving the data processing efficiency and computing performance of the calculation graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a method for compiling and optimizing a computational graph provided in Embodiment 1 of the present invention;

[0017] Figure 2 It is a schematic diagram of the structure of the calculation graph provided in the first embodiment of the present invention;

[0018] Figure 3 It is a flowchart of a method for compiling and optimizing a computational graph provided in Embodiment 2 of the present invention;

[0019] Figure 4 It is a structural diagram of a computing graph compilation optimization device provided by Embodiment 3 of the present invention;

[0020] Figure 5 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0022] Embodiment 1

[0023] Figure 1 This is a flowchart of a method for compiling and optimizing a computational graph provided in the first embodiment of the present invention. This embodiment is applicable to the case of compiling and optimizing a computational graph. The method can be executed by a device for compiling and optimizing a computational graph, which can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0024] Step S101, screening the computation graph running on the multi-level storage chip to obtain candidate operator nodes.

[0025] Specifically, the computation graph of this embodiment may include multiple operator nodes, and the operator nodes represent operation logic computation operations such as add and mul, or data handling operations such as slice, etc. And the computation graph in this embodiment is run on a multi-level storage chip, for example, a multi-level storage chip including three storage layers of L1, L2 and L3, wherein L3 storage is a global storage space, which is the highest level storage layer, with the largest capacity but the slowest speed; L2 storage is a shared storage layer, which can be accessed by multiple computing units, such as SIP, with a smaller capacity but a higher speed; L1 storage is a computing unit exclusive storage, with the smallest capacity but the fastest speed, and this embodiment mainly performs the fusion of operator nodes in the shared storage layer. Among them, fusion mainly refers to IO fusion. When IO fusion is not performed, the output data of the operator nodes in the calculation graph are stored in the L3 level, which is the highest level storage layer, by default. The operator loads data from L3 to L2 during calculation, and then each calculation unit SIP loads a part of the data to L1 for calculation. After the calculation is completed, it is finally written back to L3 from L1 through L2. This operator node is executed, and then the next operator node starts to load data from L3. This method will obviously increase the number of data transmissions at different storage levels. When IO fusion is used, the output of the previous operator node will not be written back to L3, but temporarily stored in L2, so the next operator node can directly load data from L2, that is, only the previous operator node loads data from L3, and the next operator node writes the calculation results after the continuous execution of the two operator logics back to L3. At this time, it is considered that the two operator nodes are fused into a fusion operator node. Of course, in this embodiment, only a multi-level storage chip containing three storage layers is used as an example for explanation, and the specific form of the multi-level storage chip is not limited.

[0026] Optionally, a computation graph running on a multi-level storage chip is screened to obtain candidate operator nodes, including: obtaining the residency status of each operator node in the computation graph in a shared storage layer, wherein the residency status includes having the residency condition and not having the residency condition; querying the support status of the input or output of each operator node in the computation graph in the shared storage layer by calling a query interface, wherein the support status includes supported and not supported; and taking the operator nodes whose residency status is having the residency condition and whose support status is supported as candidate operator nodes.

[0027] Optionally, obtaining the residency status of each operator node in the computation graph in the shared storage layer includes: obtaining the tensor size of each operator node in the computation graph and the storage capacity of the shared storage layer; determining whether the tensor size of each operator node is smaller than the storage capacity, and if so, determining that the residency status of the operator node in the shared storage layer is qualified for residency; otherwise, determining that the residency status of the operator node in the shared storage layer is not qualified for residency.

[0028] Specifically, in this embodiment, before fusion is performed, the candidate operator nodes in the calculation graph that meet the fusion conditions need to be screened out, and when screening, it is mainly to determine whether the operator node has the conditions for residence in the shared storage layer, and whether it supports input or output in the shared storage layer. When it is determined that both of the above conditions are met, the operator node is determined as a candidate operator node. Among them, when judging whether the operator node has the conditions for residence in the shared storage layer, it is necessary to obtain the tensor size of the operator node and the storage capacity of the shared storage layer. When the tensor size is less than the storage capacity, it means that the storage capacity of the shared storage layer can carry the calculation process of the operator node, and at this time, it means that the residence state of the operator node in the shared storage layer is to meet the conditions for residence; when the tensor size is greater than the storage capacity, it means that the storage capacity of the shared storage layer is insufficient to carry the calculation process of the operator node, and at this time, it means that the residence state of the operator node in the shared storage layer is not to meet the conditions for residence. Since this application is to perform fusion in the shared storage layer, it is necessary to screen out the operator nodes that have the conditions for residence in the shared storage layer. In addition, when judging whether the operator node supports input or output in the shared memory layer, it is mainly queried by calling the query interface. Since the support status of the input or output of each operator node in the shared memory layer has been pre-marked, it can be queried only through the query interface without additional calculations, where the support status includes support and non-support. In this embodiment, the operator nodes that meet the above two conditions are screened out and marked as candidate operator nodes in the calculation graph. The number of candidate operator nodes can be the same as the number of operator nodes included in the calculation graph, that is, all the operator nodes in the calculation graph meet the fusion conditions, or the number of candidate operator nodes is less than the number of operator nodes included in the calculation graph, that is, some of the operator nodes in the calculation graph meet the fusion conditions. Of course, this embodiment is only an example, and does not limit the number of selected candidate operator nodes.

[0029] It should be noted that in this embodiment, when calculating the tensor size of each operator node, different calculation methods can be adopted according to the requirements of the storage content. For example, the sum of the non-constant input tensor size and the output tensor size is used as the tensor size of the operator node; or, the smallest of the non-constant input tensor size and the output tensor size is used as the tensor size of the operator node. Of course, this embodiment is only an example for illustration, and does not limit the specific calculation method of the tensor size of each operator node, which can be determined according to the different requirements of the storage content.

[0030] Step S102, obtaining the dynamic type of each current candidate operator node.

[0031] Optionally, the dynamic type of each current candidate operator node is obtained, including: obtaining the current number of non-constant input tensors of each current candidate operator node, and using the number of non-constant input tensors as the in-degree of the candidate operator node; obtaining the current number of all output tensors of each current candidate operator node, and using the number of all output tensors as the out-degree of the candidate operator node; determining the dynamic type of each current candidate operator node based on the in-degree and out-degree.

[0032] Specifically, in this embodiment, after determining the candidate operator, the dynamic type of each candidate operator node will be obtained, wherein the dynamic type includes single input single output, single input multiple output, multiple input single output and multiple input multiple output, and the dynamic type of each candidate operator node is determined according to the in-degree and out-degree. Among them, the in-degree is mainly determined according to the input parameters of the candidate operator node, and the input parameters mainly refer to the number of non-constant input tensors; the out-degree is determined according to the output parameters of the candidate operator node, and the output parameters mainly refer to the number of all output tensors, and the out-degree of this embodiment is specifically the number of other operators that depend on the output result of the current candidate operator node. For example, when it is determined that the in-degree of the candidate operator node is 1 and the out-degree is 2, the dynamic type of the candidate operator node is determined to be single input multiple output. When the in-degree of the candidate operator node is 2 and the out-degree is 2, the dynamic type of the candidate operator node is determined to be multiple input multiple output. Of course, this embodiment is only an example, and does not limit the specific values ​​of the in-degree and out-degree of the candidate operator node. Furthermore, in this embodiment, the dynamic type of each candidate operator node can be updated and changed according to the fusion result during the subsequent fusion process.

[0033] It should be noted that in this embodiment, only the number of non-constant input tensors is considered when performing in-degree calculation. This is because for the operator node, the input of the constant has been generated before the runtime. For example, when calculating the addition of two numbers, x+y is addition, and x+1 is also addition, but when adding 1, 1 is already a certain constant, and y is not known yet. The previous operator node is required to give the result of y. From this, it can be seen that the tensor data of the constant does not depend on the result calculated by the previous operator node, so there is no dependent edge in the calculation graph, and it is not included in the in-degree.

[0034] Step S103, forward search and backward search are performed on the candidate operator nodes in the calculation graph according to the dynamic type to obtain a fused operator node, and the calculation graph is updated according to the fused operator node to obtain an optimized calculation graph.

[0035] Optionally, forward search and backward search are performed on candidate operator nodes in the calculation graph according to the dynamic type to obtain fusion operator nodes, and the calculation graph is updated according to the fusion operator nodes to obtain an optimized calculation graph, including: using the specified operator node in the calculation graph as a forward search anchor point, using a greedy algorithm to forward search and fuse with reference to the dynamic type to obtain at least one fusion operator node, wherein the specified operator node includes a candidate operator node or a fusion operator node; updating the calculation graph according to the fusion operator node to obtain an updated calculation graph; using the single-input multiple-output and multiple-input multiple-output specified operator nodes in the updated calculation graph as backward search anchor points, using a greedy algorithm to backward search with reference to the dynamic type to obtain supplementary fusion operator nodes; updating the update calculation graph according to the supplementary fusion operator nodes to obtain an optimized calculation graph.

[0036] Specifically, after marking the candidate operator nodes on the computational graph, the present embodiment can traverse the candidate operator nodes to search forward and fuse them to obtain the fused operator nodes, and update the computational graph according to the fused operator nodes to obtain the updated computational graph. After obtaining the updated computational graph, the candidate operator nodes will be traversed again or the fused operator nodes obtained by fusion will be searched backward to supplement the candidate operator nodes missed by the fusion during the forward search process to obtain the supplementary fused operator nodes, and the updated computational graph will be updated again according to the supplementary fused operator nodes to obtain the optimized computational graph. Through the forward search and backward supplementary search process, as many candidate operator nodes that meet the fusion requirements as possible can be fused, wherein all candidate operator nodes in the fused operator nodes carry non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip, so the number of tensor data transfers between different storage levels in the optimized computational graph finally obtained will be significantly reduced.

[0037] Optionally, a greedy algorithm is used to forward search and fuse with reference to the dynamic type to obtain at least one fusion operator node, including: forward searching for a single-input single-output or multi-input single-output designated operator node to be fused that has a dependency relationship with the current designated operator node; obtaining the peak memory usage of the designated operator node to be fused and the forward search anchor point in the shared storage layer; and determining the fusion operator node by a greedy algorithm based on the peak memory usage and the storage capacity of the shared storage layer.

[0038] Optionally, a greedy algorithm is used to determine the fusion operator node based on the peak memory usage and the storage capacity of the shared storage layer, including: judging whether the peak memory usage exceeds the storage capacity of the shared storage layer; if so, sequentially adjusting the designated operator nodes to be fused that have no dependency relationship, and obtaining the fusion operator node based on the adjustment result; otherwise, fusing the designated operator node to be fused with the forward search anchor point to obtain an intermediate fusion operator node, updating the dynamic type of the intermediate fusion operator node, using the intermediate fusion operator node as a new forward search anchor point to increase the search range and search again, and obtaining the fusion operator node based on the result of the re-search.

[0039] Optionally, a fused operator node is obtained based on the adjustment result, including: determining whether there is an adjustment result in which the memory usage peak does not exceed the storage capacity of the shared storage layer; if so, obtaining the operator node sequence corresponding to the adjustment result, and fusing the specified operator node to be fused and the anchor point according to the operator node sequence to obtain the fused operator node; otherwise, abandoning the fusion of the anchor point and the specified operator node to be fused.

[0040] In a specific implementation, Figure 2The figure shows a schematic diagram of the structure of the computation graph, which includes five operator nodes A, B, C, D and E. Each operator node is screened to be fused, so the five operator nodes are marked as candidate operator nodes. First, when performing a forward search, the designated operator node in the computation graph is used as the forward search anchor point, and the designated operator node includes a candidate operator node or a fusion operator node. For example, A is used as the anchor point to search forward for a single-input single-output or multi-input single-output designated operator node to be fused that has a dependency relationship with A, such as B. Then, the peak memory usage a of A and B in the shared storage layer is calculated. When it is determined that the peak memory usage a does not exceed the storage capacity of the shared storage layer, A and B are directly fused to obtain the intermediate fusion operator node A2 (AB). Then, the search range is expanded again to continue searching for a single-input single-output or multi-input single-output designated operator node to be fused that has a dependency relationship with A2 (AB), such as C. Then, the peak memory usage b of ABC in the shared storage layer is calculated. When it is determined that b exceeds the memory capacity of the shared storage layer, the non-dependent designated operator node to be fused is fused. The specified operator nodes are fused and adjusted in order. For example, when it is determined that both B and C depend on A, the position of A is kept unchanged, the order of C and B is adjusted, and then the peak memory usage b' of ACB in the shared storage layer is calculated. When it is determined that b' does not exceed the memory capacity of the shared storage layer, A2 (AB) is fused with C to obtain the intermediate fusion operator node A3 (ACB), and then the search range is expanded again to continue searching, looking for a single-input single-output or multi-input multi-output specified operator node to be fused that has a dependency relationship with A3 (ACB); however, when it is determined that b' still exceeds the memory capacity of the shared storage layer, the fusion of A2 (AB) and C is abandoned, and the intermediate fusion operator node A2 (AB) is used as the fusion operator node obtained by the current search, and C is used as a new forward search anchor point to re-carry out the fusion search in the above manner. For example, after searching the calculation graph in the above forward search method, the fusion operator node obtained includes A2 (AB), and the updated calculation graph obtained by updating the calculation graph according to the fusion operator node includes four operator nodes A2 (AB), C, D and E.

[0041] It should be noted that within the fusion operator node, the usage of the shared storage layer is different when executing different operator nodes, and the peak memory usage is determined by the maximum usage time, and the order of the operator nodes determines the temporary storage data, which in turn affects the maximum time of memory usage. The peak memory usage corresponding to different operator node orders may be different. Therefore, when the peak memory usage of the fusion operator node exceeds the storage capacity of the shared storage layer, the peak memory usage of the operator node can be adjusted by adjusting the order of the operator nodes that have no dependency relationship within the fusion operator node.

[0042] Optionally, a greedy algorithm is used to perform a backward search with reference to the dynamic type to obtain a supplementary fusion operator node, including: backward searching for a single-input single-output or single-input multiple-output designated operator node to be fused that has a dependency relationship with the current operator node; obtaining the peak memory usage of the designated operator node to be fused and the backward search anchor point in the shared storage layer; and determining the supplementary fusion operator node by a greedy algorithm based on the peak memory usage and the storage capacity of the shared storage layer.

[0043] Specifically, after obtaining the updated calculation graph through forward search, in order to avoid omissions in the search of fusion operator nodes, the present embodiment will continue to search in a backward search manner. When performing the backward search, the designated operator nodes of single-input multiple-output and multiple-input multiple-output are used as the backward search anchor points. The designated operator nodes here can be candidate operator nodes that have not been fused or fusion operator nodes obtained through fusion. Then, a backward search is performed for single-input single-output or single-input multiple-output candidate operator nodes to be fused or fusion operator nodes that have a dependency relationship with the current operator node. In addition, the greedy algorithm is still used when performing the backward search. Therefore, the backward search method is roughly the same as the forward search, and will not be repeated in this embodiment. For example, the supplementary fusion operator node obtained by backward search is D3 (CDE), and the update calculation graph is updated according to the supplementary fusion operator node to obtain the optimized calculation graph. Therefore, the optimized calculation graph includes two operator nodes A2 (AB) and D3 (CDE). Of course, this implementation is only an example and does not limit the specific form of the obtained optimized calculation graph.

[0044] This implementation method screens out candidate operator nodes that meet the fusion conditions and the dynamic type of each candidate operator node from the computation graph, and searches the candidate operator nodes according to the dynamic type to obtain the fusion operator node. Since the candidate operator nodes located in the same fusion operator node only perform tensor transfer in the shared storage layer, the number of times tensor data is transferred between different storage layers is avoided, thereby improving the data processing efficiency and computing performance of the computation graph.

[0045] Embodiment 2

[0046] Figure 3 A flowchart of a method for compiling and optimizing a computational graph provided in the second embodiment of the present invention. This embodiment is based on the above embodiment. After updating the computational graph according to the fusion operator node to obtain the optimized computational graph, it also includes: uniformly registering an operator interface for the fusion operator nodes contained in the optimized computational graph, and configuring the attribute information of the fusion operator nodes through the operator interface. When an execution instruction for the optimized computational graph is received, the optimized computational graph is parsed and executed by calling the operator interface. Figure 3 As shown, the method includes:

[0047] Step S201, screening the computation graph running on the multi-level storage chip to obtain candidate operator nodes.

[0048] Optionally, a computation graph running on a multi-level storage chip is screened to obtain candidate operator nodes, including: obtaining the residency status of each operator node in the computation graph in a shared storage layer, wherein the residency status includes having the residency condition and not having the residency condition; querying the support status of the input or output of each operator node in the computation graph in the shared storage layer by calling a query interface, wherein the support status includes supported and not supported; and taking the operator nodes whose residency status is having the residency condition and whose support status is supported as candidate operator nodes.

[0049] Optionally, obtaining the residency status of each operator node in the computation graph in the shared storage layer includes: obtaining the tensor size of each operator node in the computation graph and the storage capacity of the shared storage layer; determining whether the tensor size of each operator node is smaller than the storage capacity, and if so, determining that the residency status of the operator node in the shared storage layer is qualified for residency; otherwise, determining that the residency status of the operator node in the shared storage layer is not qualified for residency.

[0050] Step S202, obtaining the dynamic type of each current candidate operator node.

[0051] Optionally, the dynamic type of each current candidate operator node is obtained, including: obtaining the current number of non-constant input tensors of each current candidate operator node, and using the number of non-constant input tensors as the in-degree of the candidate operator node; obtaining the current number of all output tensors of each current candidate operator node, and using the number of all output tensors as the out-degree of the candidate operator node; determining the dynamic type of each current candidate operator node based on the in-degree and out-degree.

[0052] Step S203, forward search and backward search are performed on the candidate operator nodes in the calculation graph according to the dynamic type to obtain a fused operator node, and the calculation graph is updated according to the fused operator node to obtain an optimized calculation graph.

[0053] Optionally, forward search and backward search are performed on candidate operator nodes in the calculation graph according to the dynamic type to obtain fusion operator nodes, and the calculation graph is updated according to the fusion operator nodes to obtain an optimized calculation graph, including: using the specified operator node in the calculation graph as a forward search anchor point, using a greedy algorithm to forward search and fuse with reference to the dynamic type to obtain at least one fusion operator node, wherein the specified operator node includes a candidate operator node or a fusion operator node; updating the calculation graph according to the fusion operator node to obtain an updated calculation graph; using the single-input multiple-output and multiple-input multiple-output specified operator nodes in the updated calculation graph as backward search anchor points, using a greedy algorithm to backward search with reference to the dynamic type to obtain supplementary fusion operator nodes; updating the update calculation graph according to the supplementary fusion operator nodes to obtain an optimized calculation graph.

[0054] Optionally, a greedy algorithm is used to forward search and fuse with reference to the dynamic type to obtain at least one fusion operator node, including: forward searching for a single-input single-output or multi-input single-output designated operator node to be fused that has a dependency relationship with the current designated operator node; obtaining the peak memory usage of the designated operator node to be fused and the forward search anchor point in the shared storage layer; and determining the fusion operator node by a greedy algorithm based on the peak memory usage and the storage capacity of the shared storage layer.

[0055] Optionally, a greedy algorithm is used to determine the fusion operator node based on the peak memory usage and the storage capacity of the shared storage layer, including: judging whether the peak memory usage exceeds the storage capacity of the shared storage layer; if so, sequentially adjusting the designated operator nodes to be fused that have no dependency relationship, and obtaining the fusion operator node based on the adjustment result; otherwise, fusing the designated operator node to be fused with the forward search anchor point to obtain an intermediate fusion operator node, updating the dynamic type of the intermediate fusion operator node, using the intermediate fusion operator node as a new forward search anchor point to increase the search range and search again, and obtaining the fusion operator node based on the result of the re-search.

[0056] Optionally, a fused operator node is obtained based on the adjustment result, including: determining whether there is an adjustment result in which the memory usage peak does not exceed the storage capacity of the shared storage layer; if so, obtaining the operator node sequence corresponding to the adjustment result, and fusing the specified operator node to be fused and the anchor point according to the operator node sequence to obtain the fused operator node; otherwise, abandoning the fusion of the anchor point and the specified operator node to be fused.

[0057] Optionally, a greedy algorithm is used to perform a backward search with reference to the dynamic type to obtain a supplementary fusion operator node, including: backward searching for a single-input single-output or single-input multiple-output designated operator node to be fused that has a dependency relationship with the current operator node; obtaining the peak memory usage of the designated operator node to be fused and the backward search anchor point in the shared storage layer; and determining the supplementary fusion operator node by a greedy algorithm based on the peak memory usage and the storage capacity of the shared storage layer.

[0058] Step S204: uniformly register an operator interface for the fusion operator nodes included in the optimization calculation graph, and configure the attribute information of the fusion operator nodes through the operator interface.

[0059] Specifically, in this embodiment, after the optimized calculation graph is obtained by fusion optimization, it is necessary to parse it first when the optimized calculation graph needs to be executed. Generally, the customized fusion is usually parsed manually, but for the automatically generated fusion operator nodes of this application, the number, structure, memory operator type and data type are not fixed, so it is necessary to parse it in an automated way. In the embodiment of this application, in order to facilitate the implementation of subsequent automated parsing, all fusion operator nodes contained in the optimized calculation graph will be uniformly registered with an operator interface, such as AutoFusionOp. Of course, this embodiment is only an example and does not limit the specific format of the operator interface. In addition, the fusion operator node and the ordinary operator node are at the same level for the framework side scheduling and distribution. The upper framework does not need to manage the inner storage, that is, the shared storage layer L2, which simplifies the framework processing. In this embodiment, after the operator interface registration is completed, the attribute information of the fusion operator node will be configured through the operator interface, wherein the attribute information includes the execution order, execution parameters, and the storage memory space of the input tensor and the output tensor.

[0060] Among them, the execution order refers to the fusion order of each operator node sorted out during fusion of the fusion operator node; since the complete operator expression is retained during fusion, the execution parameters can be extracted during parsing and integrated into the context parameter information of a single operator node, realizing the basis for calling a single operator node in the fusion operator node; if the input tensor of the operator comes from the previous operator inside the fusion, it is configured as the inner storage managed by AutoFusionOp; if it comes from outside the fusion, it is configured as the outer storage uniformly managed by the framework side. In addition, if the output tensor of the operator is not the final output of the fusion operator, it is configured to be written back to the inner storage managed by AutoFusionOp; if it is the final output of the fusion operator, it is configured to be written back to the outer storage uniformly managed by the framework side. Through the above configuration, the upper-level framework can only manage the outer storage, and the operator side and the underlying scheduler still manage the inner storage, completing the adaptation of the software stack at the lowest cost.

[0061] Step S205: When an execution instruction for the optimization calculation graph is received, the optimization calculation graph is parsed and executed by calling the operator interface.

[0062] Optionally, the optimization calculation graph is parsed and executed by calling the operator interface, including: decomposing the fusion operator nodes in the optimization calculation graph by calling the operator interface to obtain candidate operator nodes, and obtaining attribute information of the candidate operator nodes by calling the operator interface; executing the candidate operator nodes according to the execution parameters based on the execution order, and saving the execution results of each candidate operator node in the corresponding memory space.

[0063] Specifically, in this implementation, when an execution instruction for an optimized computational graph is received, the operator interface will be called, and all automatically fused operator nodes will enter the operator interface to complete the calculation, and the fused operator nodes will be disassembled in the operator interface to obtain candidate operator nodes. In addition, the attribute information of each pre-configured candidate operator node will be obtained through the operator interface, and the candidate operator nodes will be executed in the execution order based on the execution parameters. The execution results of each candidate operator node will be saved accordingly for the input tensors and output tensors according to the storage memory space configured in the operator interface.

[0064] It is worth mentioning that for operator nodes that have not been fused, each call requires a dispatch action to be executed, which brings corresponding fixed overhead. However, in this application, by calling the operator interface to parse the fused operator node, only one dispatch action is required to achieve the dispatch of multiple operator nodes, thereby eliminating the excessive fixed overhead caused by multiple dispatches. This has a significant optimization effect for operators with short execution time. In addition, the life cycle analysis of temporary result data stored in the inner storage can grasp the application and release of the inner storage as a whole and make full use of the inner storage space.

[0065] This implementation method screens out candidate operator nodes that meet the fusion conditions and the dynamic type of each candidate operator node from the computation graph, and searches the candidate operator nodes according to the dynamic type to obtain the fusion operator node. Since the candidate operator nodes located in the same fusion operator node only perform tensor transfer in the shared storage layer, the number of times tensor data is transferred between different storage layers is avoided, thereby improving the data processing efficiency and computing performance of the computation graph.

[0066] Embodiment 3

[0067] Figure 4 This is a schematic diagram of the structure of a computing graph compilation optimization device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a candidate operator node screening module 310, a dynamic type acquisition module 320 and an optimized calculation graph acquisition module 330.

[0068] The candidate operator node screening module 310 is used to screen the computation graph running on the multi-level storage chip to obtain candidate operator nodes;

[0069] The dynamic type acquisition module 320 is used to acquire the dynamic type of each current candidate operator node, wherein the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output and multiple-input multiple-output;

[0070] The optimized calculation graph acquisition module 330 is used to perform forward search and backward search on the candidate operator nodes in the calculation graph according to the dynamic type to obtain the fusion operator node, and update the calculation graph according to the fusion operator node to obtain the optimized calculation graph.

[0071] Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

[0072] Optionally, a candidate operator node screening module is used to obtain the residency status of each operator node in the computation graph in the shared storage layer, wherein the residency status includes meeting the residency conditions and not meeting the residency conditions;

[0073] Query the support status of the input or output of each operator node in the computation graph in the shared storage layer by calling the query interface, where the support status includes supported and unsupported;

[0074] The operator nodes whose resident status is qualified for resident and whose support status is supported are taken as candidate operator nodes.

[0075] Optionally, the candidate operator node screening module is also used to obtain the tensor size of each operator node in the computation graph and the storage capacity of the shared storage layer;

[0076] Determine whether the tensor size of each operator node is less than the storage capacity. If so, determine whether the operator node's residency status in the shared storage layer meets the residency conditions.

[0077] Otherwise, it is determined that the residency state of the operator node in the shared storage layer is not in a residency condition.

[0078] Optionally, a dynamic type acquisition module is used to obtain the current number of non-constant input tensors of each candidate operator node, and use the number of non-constant input tensors as the in-degree of the candidate operator node;

[0079] Get the total number of output tensors of each candidate operator node, and use the total number of output tensors as the out-degree of the candidate operator node;

[0080] The dynamic type of each current candidate operator node is determined based on the in-degree and out-degree.

[0081] Optionally, the optimized calculation graph acquisition module includes: a forward search unit, which is used to use a specified operator node in the calculation graph as a forward search anchor point, adopt a greedy algorithm to forward search and fuse to obtain at least one fusion operator node with reference to the dynamic type, wherein the specified operator node includes a candidate operator node or a fusion operator node;

[0082] An update calculation graph acquisition unit, used to update the calculation graph according to the fusion operator node to obtain an update calculation graph;

[0083] A backward search unit is used to update the designated SIMO and MIMO operator nodes in the computation graph as backward search anchor points, and adopt a greedy algorithm to perform backward search with reference to the dynamic type to obtain supplementary fusion operator nodes;

[0084] The optimized computation graph acquisition unit is used to update the updated computation graph according to the supplemented fusion operator nodes to acquire the optimized computation graph.

[0085] Optionally, a forward search unit, used to forward search for a single-input single-output or multi-input single-output designated operator node to be fused that has a dependency relationship with the current designated operator node;

[0086] Get the peak memory usage of the specified operator node to be fused and the forward search anchor point in the shared storage layer;

[0087] A greedy algorithm is used to determine the fusion operator nodes based on the peak memory usage and the storage capacity of the shared storage layer.

[0088] Optionally, the forward search unit is further used to determine whether the peak memory usage exceeds the storage capacity of the shared storage layer. If so, the designated operator nodes to be fused that do not have dependencies are sequentially adjusted, and the fused operator nodes are obtained according to the adjustment results.

[0089] Otherwise, the specified operator node to be fused is fused with the forward search anchor point to obtain an intermediate fusion operator node, the dynamic type of the intermediate fusion operator node is updated, the intermediate fusion operator node is used as a new forward search anchor point to increase the search range and search again, and the fusion operator node is obtained according to the result of the re-search.

[0090] Optionally, the forward search unit is further used to determine whether there is an adjustment result in which the peak value of memory usage does not exceed the storage capacity of the shared storage layer. If so, the operator node sequence corresponding to the adjustment result is obtained, and the specified operator node to be fused and the anchor point are fused according to the operator node sequence to obtain a fused operator node.

[0091] Otherwise, the fusion of the anchor point and the specified operator node to be fused is abandoned.

[0092] Optionally, a backward search unit, used to backward search for a single-input single-output or single-input multiple-output designated operator node to be fused that has a dependency relationship with the current operator node;

[0093] Get the peak memory usage of the specified operator node to be fused and the backward search anchor point in the shared storage layer;

[0094] A greedy algorithm is used to determine the supplementary fusion operator nodes based on the peak memory usage and the storage capacity of the shared storage layer.

[0095] Optionally, the device further includes an operator interface registration module, which is used to uniformly register an operator interface for the fusion operator nodes included in the optimization calculation graph, and configure the attribute information of the fusion operator nodes through the operator interface, wherein the attribute information includes the execution order, execution parameters, and the storage memory space of the input tensor and the output tensor;

[0096] The parsing and execution module is used to parse and execute the optimized calculation graph by calling the operator interface when receiving the execution instruction for the optimized calculation graph.

[0097] Optionally, the parsing execution module is also used to disassemble the fusion operator nodes in the optimization calculation graph by calling the operator interface to obtain candidate operator nodes.

[0098] Obtain the attribute information of the candidate operator node by calling the operator interface;

[0099] The candidate operator nodes are executed according to the execution parameters based on the execution order, and the execution results of each candidate operator node are saved in the corresponding memory space.

[0100] The computation graph compilation optimization device provided in the embodiment of the present invention can execute the computation graph compilation optimization method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0101] Embodiment 4

[0102] Figure 5 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention is shown in FIG. Figure 5 As shown, the computer device includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of the processor 610 in the computer device can be one or more. Figure 5 A processor 610 is taken as an example; the processor 610, memory 620, input device 630 and output device 640 in the computer device can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0103] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for compiling and optimizing the computational graph in the embodiment of the present invention. The processor 610 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 620, that is, implements the above-mentioned method for compiling and optimizing the computational graph, including:

[0104] Screen the computational graph running on the multi-level storage chip to obtain candidate operator nodes;

[0105] Get the dynamic type of each current candidate operator node, where the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output, and multiple-input multiple-output;

[0106] According to the dynamic type, the candidate operator nodes in the calculation graph are forward searched and backward searched to obtain the fusion operator nodes, and the calculation graph is updated according to the fusion operator nodes to obtain the optimized calculation graph.

[0107] Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

[0108] The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 620 may further include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] The input device 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 640 may include a display device such as a display screen.

[0110] Embodiment 5

[0111] Embodiment 5 of the present invention further provides a storage medium including computer executable instructions, which are used to execute a method for compiling and optimizing a computational graph when executed by a computer processor, including:

[0112] Screen the computational graph running on the multi-level storage chip to obtain candidate operator nodes;

[0113] Get the dynamic type of each current candidate operator node, where the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output, and multiple-input multiple-output;

[0114] According to the dynamic type, the candidate operator nodes in the calculation graph are forward searched and backward searched to obtain the fusion operator nodes, and the calculation graph is updated according to the fusion operator nodes to obtain the optimized calculation graph.

[0115] Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

[0116] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention, whose computer executable instructions are not limited to the above method operations, can also execute related operations in the compilation optimization method of the computational graph provided by any embodiment of the present invention.

[0117] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0118] It is worth noting that in the above-mentioned embodiment of the compilation optimization device according to the computational graph, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0119] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for compiling and optimizing a computational graph, characterized in that: include: Screen the computational graph running on the multi-level storage chip to obtain candidate operator nodes; Obtaining the dynamic type of each of the current candidate operator nodes, wherein the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output, and multiple-input multiple-output; According to the dynamic type, forward search and backward search are performed on the candidate operator nodes in the calculation graph to obtain a fusion operator node, and according to the fusion operator node, the calculation graph is updated to obtain an optimized calculation graph, Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

2. The method according to claim 1, characterized in that: The step of screening the computation graph running on the multi-level storage chip to obtain candidate operator nodes includes: Obtaining the residency status of each operator node in the computation graph in the shared storage layer, wherein the residency status includes having a residency condition and not having a residency condition; Query the support status of the input or output of each operator node in the computation graph in the shared storage layer by calling the query interface, wherein the support status includes supported and unsupported; The operator node whose resident status is satisfied with the resident condition and whose support status is supported is taken as the candidate operator node.

3. The method according to claim 2, characterized in that The obtaining of the resident status of each operator node in the computation graph in the shared storage layer includes: Obtaining the tensor size of each operator node in the computation graph and the storage capacity of the shared storage layer; Determine whether the tensor size of each operator node is smaller than the storage capacity, and if so, determine that the resident state of the operator node in the shared storage layer is the resident condition. Otherwise, it is determined that the residency state of the operator node in the shared storage layer does not meet the residency condition.

4. The method according to claim 1, characterized in that The obtaining of the dynamic type of each of the current candidate operator nodes includes: Obtain the current number of non-constant input tensors of each of the candidate operator nodes, and use the number of non-constant input tensors as the in-degree of the candidate operator node; Obtain the current total number of output tensors of each of the candidate operator nodes, and use the total number of output tensors as the out-degree of the candidate operator node; The dynamic type of each of the current candidate operator nodes is determined according to the in-degree and the out-degree.

5. The method according to claim 1, characterized in that: The step of performing forward search and backward search on the candidate operator nodes in the computation graph according to the dynamic type to obtain a fused operator node, and updating the computation graph according to the fused operator node to obtain an optimized computation graph includes: Taking the designated operator node in the computation graph as a forward search anchor point, a greedy algorithm is used to forward search and fuse to obtain at least one fused operator node with reference to the dynamic type, wherein the designated operator node includes a candidate operator node or a fused operator node; Updating the computation graph according to the fusion operator node to obtain an updated computation graph; Using the designated SIMO and MIMO operator nodes in the update calculation graph as backward search anchor points, and using the greedy algorithm to perform backward search with reference to the dynamic type to obtain a supplementary fusion operator node; The update calculation graph is updated according to the supplementary fusion operator node to obtain the optimized calculation graph.

6. The method according to claim 5, characterized in that The step of using a greedy algorithm to search forward with reference to the dynamic type and fusing to obtain at least one fusion operator node includes: Search forward for a single-input single-output or multi-input single-output designated operator node to be fused that has a dependency relationship with the current designated operator node; Obtaining the peak memory usage of the designated operator node to be fused and the forward search anchor point in the shared storage layer; The fusion operator node is determined by using the greedy algorithm according to the memory usage peak and the storage capacity of the shared storage layer.

7. The method according to claim 6, characterized in that The step of determining the fusion operator node by using the greedy algorithm according to the memory usage peak value and the storage capacity of the shared storage layer includes: Determine whether the peak memory usage exceeds the storage capacity of the shared storage layer. If so, sequentially adjust the designated operator nodes to be fused that do not have a dependency relationship, and obtain the fused operator node according to the adjustment result. Otherwise, the designated operator node to be fused is fused with the forward search anchor point to obtain an intermediate fusion operator node, the dynamic type of the intermediate fusion operator node is updated, the intermediate fusion operator node is used as a new forward search anchor point to increase the search range and search again, and the fusion operator node is obtained according to the result of the re-search.

8. The method according to claim 7, characterized in that The obtaining the fusion operator node according to the adjustment result includes: Determine whether there is an adjustment result in which the peak value of memory usage does not exceed the storage capacity of the shared storage layer. If so, obtain the operator node sequence corresponding to the adjustment result, and fuse the designated operator node to be fused and the anchor point according to the operator node sequence to obtain the fused operator node. Otherwise, the fusion of the anchor point and the designated operator node to be fused is abandoned.

9. The method according to claim 5, characterized in that The step of using a greedy algorithm to search backward with reference to the dynamic type to obtain a supplementary fusion operator node includes: Search backward for a single-input single-output or single-input multiple-output designated operator node to be fused that has a dependency relationship with the current operator node; Obtaining the peak memory usage of the designated operator node to be fused and the backward search anchor point in the shared storage layer; The supplementary fusion operator node is determined by using the greedy algorithm according to the memory usage peak and the storage capacity of the shared storage layer.

10. The method according to claim 1, characterized in that After the calculation graph is updated according to the fusion operator node to obtain the optimized calculation graph, the method further includes: Registering an operator interface uniformly for the fusion operator nodes included in the optimization calculation graph, and configuring the attribute information of the fusion operator nodes through the operator interface, wherein the attribute information includes execution order, execution parameters, and storage memory space of input tensors and output tensors; When an execution instruction for the optimization calculation graph is received, the optimization calculation graph is parsed and executed by calling the operator interface.

11. The method according to claim 10, characterized in that The parsing and executing the optimization calculation graph by calling the operator interface includes: By calling the operator interface, the fusion operator node in the optimization calculation graph is disassembled to obtain candidate operator nodes, Acquire the attribute information of the candidate operator node by calling the operator interface; The candidate operator nodes are executed according to the execution parameters based on the execution order, and the execution results of each candidate operator node are saved in a corresponding memory space.

12. A computing graph compilation optimization device, characterized in that: include: A candidate operator node screening module is used to screen the computation graph running on the multi-level storage chip to obtain candidate operator nodes; A dynamic type acquisition module, used to acquire the dynamic type of each of the current candidate operator nodes, wherein the dynamic type includes single-input single-output, single-input multiple-output, multiple-input single-output and multiple-input multiple-output; An optimized calculation graph acquisition module is used to perform forward search and backward search on the candidate operator nodes in the calculation graph according to the dynamic type to obtain a fusion operator node, and update the calculation graph according to the fusion operator node to obtain an optimized calculation graph, Among them, all candidate operator nodes in the fusion operator node carry out the transfer of non-constant input tensors and output tensors in the shared storage layer of the multi-level storage chip.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 11 is implemented.

14. A computer executable instruction storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images