Computational graph optimization method and device, storage medium and program product

Abstract

The embodiment of the invention provides a calculation graph optimization method and device, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: segmenting a calculation graph of a coding module into a plurality of calculation sub-graphs based on a preset template; obtaining sub-graph merging granularity based on the resource quantity of the index resources on the artificial intelligence chip; and then, according to the sub-graph merging granularity, assembling and merging the plurality of calculation sub-graphs to obtain at least one merged sub-graph, that is, under the condition of fully considering index resources on an artificial intelligence chip, the calculation graph is segmented into the merged sub-graphs as few as possible, so that when each merged sub-graph is compiled into a corresponding merged kernel function and executed, the operation efficiency is improved, and the calculation efficiency is improved. The number of combined kernel functions is greatly reduced, so that the time overhead of transmitting the kernel functions is reduced. And secondly, operators contained in the computational graph are considered, and meanwhile, the resource condition of a bottom layer chip for actually executing the computational graph is combined, so that the computational graph is optimized to the maximum extent, and the optimization effect of the computational graph is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a computational graph optimization method, device, storage medium, and program product. Background Art

[0002] In the field of deep learning, encoder modules have been widely used in many models, such as the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers). In the era of large models, the importance of encoder modules has become more prominent. Therefore, it is particularly important to optimize the computational graph of the encoder module to improve the execution performance of the encoder module.

[0003] In related technologies, the computational graph is optimized through the common graph optimization strategies in the deep learning framework. Based on the consideration of universality, these optimization strategies are often targeted at some common scenarios in deep learning models and have no connection with the underlying chips that actually perform model reasoning. Therefore, it is difficult to achieve a better optimization effect when using this method to optimize the computational graph of the encoding module. Summary of the invention

[0004] The embodiments of the present application provide a computational graph optimization method, device, storage medium, and program product for improving the optimization effect of the computational graph of a coding module, thereby improving the execution performance of the coding module.

[0005] On the one hand, an embodiment of the present application provides a computational graph optimization method, including: The computational graph of the encoding module is divided into multiple computational subgraphs based on a preset template; Based on the resource amount of index resources on the AI ​​chip, the subgraph merging granularity is obtained; According to the subgraph merging granularity, assembling and merging the multiple computation subgraphs to obtain at least one merged subgraph; Each merged subgraph is compiled into a corresponding merged kernel function, and the merged kernel function corresponding to each merged subgraph is executed to obtain a calculation result of the encoding module.

[0006] On the one hand, an embodiment of the present application provides a computational graph optimization device, including: A segmentation module, used to segment the computation graph of the encoding module into multiple computation subgraphs based on a preset template; A merging module, configured to obtain a subgraph merging granularity based on the resource amount of the index resources on the artificial intelligence chip; assemble and merge the multiple computing subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph; The execution module is used to compile each merged subgraph into a corresponding merged kernel function, and execute the merged kernel function corresponding to each merged subgraph to obtain the calculation result of the encoding module.

[0007] Optionally, the index resource is used to: index a video memory object for an input tensor or an output tensor; The merging module is specifically used for: Based on the resource amount of the index resource, determine an upper limit value of the total number of input tensors and output tensors included in a merged subgraph; The upper limit value is used as the sub-graph merging granularity.

[0008] Optionally, the merging module is specifically used for: According to the subgraph merging granularity and the execution order of the multiple computing subgraphs, the multiple computing subgraphs are assembled and merged in sequence to obtain at least one merged subgraph.

[0009] Optionally, the merging module is further used for: After assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph, the following operations are performed for each merged subgraph: Each intermediate tensor of a merged subgraph is traversed, and each time an intermediate tensor is traversed, cache resources in the artificial intelligence chip are allocated to the intermediate tensor, wherein the intermediate tensor is a tensor transferred between computational subgraphs in the merged subgraph.

[0010] Optionally, the merging module is specifically used for: Each time an intermediate tensor is traversed and it is determined that the storage space occupied by the intermediate tensor is less than a preset threshold, the cache resource is allocated to the intermediate tensor.

[0011] Optionally, the cache resources allocated to the intermediate tensor do not overlap with the cache resources allocated to other intermediate tensors between the computational subgraphs that the intermediate tensor depends on.

[0012] Optionally, the merging module is further used for: Before assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph, for each computation subgraph, if the computation subgraph includes at least one operator combination for layout adjustment, performing the following subgraph optimization operation on the computation subgraph: The at least one operator combination for layout adjustment contained in the computation subgraph is replaced with the output addresses corresponding to the at least one operator combination for layout adjustment.

[0013] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned computational graph optimization method when executing the computer program.

[0014] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned computational graph optimization method.

[0015] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned computational graph optimization method.

[0016] In an embodiment of the present application, the computational graph of the encoding module is divided into multiple computational subgraphs based on a preset template; then, based on the resource amount of the index resources on the artificial intelligence chip, the subgraph merging granularity is obtained; and then, according to the subgraph merging granularity, multiple computational subgraphs are assembled and merged to obtain at least one merged subgraph, that is, the computational graph is divided into as few merged subgraphs as possible while fully considering the index resources on the artificial intelligence chip. Therefore, when each merged subgraph is compiled into a corresponding merged kernel function and executed, the number of merged kernel functions is greatly reduced, thereby reducing the time overhead of transmitting the kernel function. Secondly, the computational graph optimization method of the present application not only takes into account the operators contained in the computational graph itself, but also combines the resource situation of the underlying chip that actually executes the computational graph, so that the computational graph is optimized to the maximum extent, thereby improving the optimization effect of the computational graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application; Figure 2 A flowchart of a computational graph optimization method provided in an embodiment of the present application; Figure 3 A calculation graph of a forward encoding of BERT provided in an embodiment of the present application; Figure 4A schematic diagram of a calculation graph segmentation result provided in an embodiment of the present application Figure 1 ; Figure 5 A calculation graph of the reverse encoding of BERT provided in an embodiment of the present application; Figure 6 A schematic diagram of a calculation graph segmentation result provided in an embodiment of the present application Figure 2 ; Figure 7 A schematic diagram of a subgraph optimization method provided in an embodiment of the present application Figure 1 ; Figure 8 A schematic diagram of a subgraph optimization method provided in an embodiment of the present application Figure 2 ; Fig. 9 A schematic diagram of a cache resource allocation method provided in an embodiment of the present application Figure 1 ; Fig.10 A schematic diagram of a cache resource allocation method provided in an embodiment of the present application Figure 2 ; Fig.11 A schematic diagram of the structure of a computational graph optimization device provided in an embodiment of the present application; Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] refer to Figure 1 , which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application, the artificial intelligence chip 100 at least includes: a video memory 101, a plurality of execution units 102, wherein each execution unit 102 includes: an on-chip cache 103, and a register 104.

[0021] The video memory 101 may be a high bandwidth memory (HBM) or other types of memory. The on-chip cache 103 is a temporary memory, which has a smaller capacity than the video memory 101 but a faster data exchange speed than the video memory 101. The on-chip cache 103 may be a general matrix main buffer (Gemm Main Buffer, GMB).

[0022] Compared with the on-chip cache 103, the register 104 has a smaller capacity than the on-chip cache 103, but a faster data exchange speed than the on-chip cache 103. The register 104 may be a thread-local register (TLR for short).

[0023] In addition to the above-mentioned structure, the artificial intelligence chip 100 in the present application may also include other structures, which are not specifically limited in the present application.

[0024] The artificial intelligence chip 100 can be: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a domain-specific architecture (DSA), etc.

[0025] In the field of deep learning, encoding modules have been widely used in many models, especially in sequence modeling and natural language processing. The encoding module in the pre-trained language model BERT is the core component of the Transformer architecture, which represents the logic composed of a series of operators. In the era of large models, the importance of encoding modules has become more and more prominent. Therefore, it is particularly important to optimize the computational graph of the encoding module to improve the execution performance of the encoding module.

[0026] In related technologies, the computational graph is optimized through the common graph optimization strategies in the deep learning framework. Based on the consideration of universality, these optimization strategies are often targeted at some common scenarios in deep learning models and have no connection with the underlying chips that actually perform model reasoning. Therefore, it is difficult to achieve a better optimization effect when using this method to optimize the computational graph of the encoding module.

[0027] In view of this, this application is based on Figure 1 The architecture diagram of the artificial intelligence chip shown provides a method for optimizing a computational graph, which mainly involves optimizing the computational graph of the encoding module. In this application, the encoding module is used to convert input data into a vector or feature representation of a fixed size. The encoding module can be applied to various scenarios. For example, image processing scenarios, speech processing scenarios, text processing scenarios, etc. In different application scenarios, the physical meaning of the input data of the encoding module may be different.

[0028] For example, in a speech processing scenario, the input data of the encoding module can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis. For example, in an image processing scenario, the input data of the encoding module can be image data used in tasks such as image preprocessing, image segmentation, and object detection. For example, in a text processing scenario, the input data of the encoding module can be text data used in tasks such as text generation and text recognition.

[0029] The following specifically describes the process of the computational graph optimization method, which is executed by a computer device, the computer device comprising Figure 1 The artificial intelligence chip shown includes the following steps: Step 201: divide the computation graph of the encoding module into multiple computation subgraphs based on a preset template.

[0030] Specifically, multiple templates (patterns) are pre-set. The templates can be some complex fusion operators, such as the fusion operator of the convolution operator and the bit-by-bit calculation operator, the fusion operator of the matrix multiplication operator and the bit-by-bit calculation operator, etc.; they can also be single operators, such as normalization operators, reduction operators, etc.; they can also be some special functions, such as loss functions.

[0031] The computation graph of the encoding module is automatically divided into multiple computation subgraphs by using the pattern matching method. In the specific implementation, the computation graph is matched with each preset template. When the computation graph contains content that matches the template, the matched content is used as a computation subgraph; for content in the computation graph that does not match any template, each original operator in the content is used as a computation subgraph.

[0032] In this way, each computation subgraph includes at least one original operator in the computation graph. At least one original operator contained in a computation subgraph constitutes a fusion operator. The number of computation subgraphs obtained after switching is less than the number of original operators contained in the computation graph.

[0033] In some embodiments, the computation graph of the encoding module is: a computation graph of a forward encoding module, or a computation graph of a reverse encoding module, wherein the structure of the computation graph of the forward encoding module is different from the structure of the computation graph of the reverse encoding module.

[0034] For example, see Figure 3, which is the calculation graph of the forward encoding module of BERT provided in the embodiment of the present application. The original operators contained in the calculation graph are: forward layer normalization (LayerNormFwd) operator, matrix multiplication (Matmul) operator, point-to-point (PointWise) operator, deformation (Reshape) operator, reordering (Permute) operator, forward normalization (SoftmaxFwd) operator, etc.

[0035] The process of performing forward encoding according to the calculation graph is: The input tensor is input into the forward layer normalization operator 1 for calculation to obtain the layer normalization result 1, wherein the input tensor can be the input data of the model or the output result of the previous module in the forward encoding module of the model.

[0036] The layer normalization result 1 is calculated in sequence using a matrix multiplication operator 1, a point-to-point operator 1, a deformation operator 1, and a reordering operator 1 to obtain a reordering result 1.

[0037] The layer normalization result 1 is calculated in sequence using the matrix multiplication operator 2, the point-to-point operator 2, the deformation operator 2, and the reordering operator 2 to obtain the reordering result 2.

[0038] The layer normalization result 1 is calculated in sequence using the matrix multiplication operator 3, the point-to-point operator 3, the deformation operator 3, and the reordering operator 3 to obtain the reordering result 3.

[0039] The reordering result 1 and the reordering result 2 are input into the matrix multiplication operator 4 for calculation to obtain the forward matrix multiplication result 1. The forward matrix multiplication result 1 is calculated in sequence using the point-to-point operator 4, the point-to-point operator 5 and the forward normalization operator to obtain the normalized result.

[0040] The normalized result and the reordering result 3 are input into the matrix multiplication operator 5 for calculation to obtain the forward matrix multiplication result 2. The forward matrix multiplication result 2 is calculated in sequence using the reordering operator 4, the deformation operator 4, the matrix multiplication operator 6, and the point-to-point operator 6 to obtain the point-to-point calculation result 1.

[0041] The layer normalization result 1 and the point-to-point calculation result 1 are input into the point-to-point operator 7 for calculation to obtain the point-to-point calculation result 2. The point-to-point calculation result 2 is input into the forward layer normalization operator 2 for calculation to obtain the layer normalization result 2.

[0042] The layer normalization result 2 is calculated in sequence using a matrix multiplication operator 7, a point-to-point operator 8, a point-to-point operator 9, a matrix multiplication operator 8, and a point-to-point operator 10 to obtain a point-to-point calculation result 3.

[0043] The layer normalization result 2 and the point calculation result 3 are input into the point-to-point operator 11 for calculation to obtain the output tensor.

[0044] Three templates are preset, namely Template 1, Template 2 and Template 3, where Template 1 is a fusion operator consisting of a matrix multiplication operator, a point-to-point operator, a deformation operator and a reordering operator; Template 2 is a fusion operator consisting of a matrix multiplication operator, a point-to-point operator and a point-to-point operator; Template 3 is a fusion operator consisting of a matrix multiplication operator, a reordering operator and a deformation operator.

[0045] Using template matching, Figure 3 The computational graph of is divided into 11 computational subgraphs (i.e., forward subgraphs), see Figure 4 , respectively: forward sub Figure 1 To the forward Fig.11 .

[0046] In the specific implementation, since the forward layer normalization operator 1 in the calculation graph does not match the above three templates, the forward layer normalization operator 1 is used as the forward sub Figure 1 .

[0047] Since the matrix multiplication operator 1, point-to-point operator 1, deformation operator 1, and reordering operator 1 in the calculation graph match template 1, the matrix multiplication operator 1, point-to-point operator 1, deformation operator 1, and reordering operator 1 in the calculation graph are divided into forward sub-operators. Figure 2 .

[0048] By analogy, other forward subgraphs can be obtained in the same way. The specific contents of other forward subgraphs are as follows: Forward Figure 3 Includes: matrix multiplication operator 2, point-to-point operator 2, deformation operator 2, reordering operator 2; Forward Figure 4 Including: matrix multiplication operator 3, point-to-point operator 3, deformation operator 3, reordering operator 3; Forward Figure 5 Including: matrix multiplication operator 4, point-to-point operator 4, point-to-point operator 5; Forward Figure 6 Includes: forward normalization operator; Forward Figure 7 Including: matrix multiplication operator 5, reordering operator 4, deformation operator 4; Forward Figure 8 Including: matrix multiplication operator 6, point-to-point operator 6, point-to-point operator 7; Forward Fig. 9 Includes: forward layer normalization operator 2; Forward Fig.10 Including: matrix multiplication operator 7, point-to-point operator 8, point-to-point operator 9; Forward Fig.11It includes: a matrix multiplication operator 8, a point-to-point operator 10, and a point-to-point operator 11.

[0049] Depend on Figure 3 and Figure 4 It can be seen that Figure 3 The computation graph of contains 30 operators, and Figure 3 After the computation graph shown is split, 11 computation subgraphs are obtained, that is, the number of computation subgraphs contained in the computation graph is less than the number of operators contained in the computation graph.

[0050] For example, see Figure 5 , which is the calculation graph of the reverse encoding module of BERT provided in the embodiment of the present application. The original operators contained in the calculation graph are: reverse layer normalization (LayerNormBwd) operator, reduction (Reduce) operator, matrix multiplication (Matmul) operator, point-to-point (PointWise) operator, deformation (Reshape) operator, reordering (Permute) operator, reverse normalization (SoftmaxFwd) operator, etc.

[0051] The process of performing reverse encoding according to the calculation graph is: The reverse layer normalization operator 1 is used to calculate the gradient information to obtain the layer normalization result 3.

[0052] The layer normalization result 3 is calculated using the reduction operator 1 to obtain the reduction result 1. The layer normalization result 3 is calculated using the matrix multiplication operator 9 to obtain the inverse matrix multiplication result 1. The layer normalization result 3 is calculated using the matrix multiplication operator 10 and the point-to-point operator 12 to obtain the point-to-point calculation result 4.

[0053] The point-to-point calculation result 4 is calculated using the reduction operator 2 to obtain the reduction result 2. The point-to-point calculation result 4 is calculated using the matrix multiplication operator 11 to obtain the inverse matrix multiplication result 2. The point-to-point calculation result 4 is calculated using the matrix multiplication operator 12 and the point-to-point operator 13 in sequence to obtain the point-to-point calculation result 5.

[0054] The point-to-point calculation result 5 is calculated using the reverse layer normalization operator 2 to obtain the layer normalization result 4.

[0055] The layer normalization result 4 is calculated by using the reduction operator 3 to obtain the reduction result 3. The layer normalization result 4 is calculated by using the matrix multiplication operator 13 to obtain the inverse matrix multiplication result 3. The layer normalization result 4 is calculated by using the matrix multiplication operator 14, the deformation operator 5, and the reordering operator 5 to obtain the reordering result 4.

[0056] The matrix multiplication operator 15, the inverse normalization operator, and the point-to-point operator 14 are used to sequentially calculate the reordering result 4 to obtain the point-to-point calculation result 6.

[0057] The matrix multiplication operator 16, the reordering operator 6, and the deformation operator 6 are used to calculate the reordering result 4 in sequence to obtain the deformation result 1.

[0058] The deformation result 1 is calculated by using the reduction operator 4 to obtain the reduction result 4. The deformation result 1 is calculated by using the matrix multiplication operator 17 to obtain the inverse matrix multiplication result 4. The deformation result 1 is calculated by using the matrix multiplication operator 18 to obtain the inverse matrix multiplication result 5.

[0059] The matrix multiplication operator 19, the reordering operator 7, and the deformation operator 7 are used to sequentially calculate the point-to-point calculation result 6 to obtain the deformation result 2.

[0060] The deformation result 2 is calculated by using the reduction operator 5 to obtain the reduction result 5. The deformation result 2 is calculated by using the matrix multiplication operator 20 to obtain the inverse matrix multiplication result 6. The deformation result 2 is calculated by using the matrix multiplication operator 21 to obtain the inverse matrix multiplication result 7.

[0061] The matrix multiplication operator 22, the reordering operator 8, and the deformation operator 8 are used to sequentially calculate the point-to-point calculation result 6 to obtain the deformation result 3.

[0062] The deformation result 3 is calculated by using the reduction operator 6 to obtain the reduction result 6. The deformation result 3 is calculated by using the matrix multiplication operator 23 to obtain the inverse matrix multiplication result 8. The deformation result 3 is calculated by using the matrix multiplication operator 24 to obtain the inverse matrix multiplication result 9.

[0063] The inverse matrix multiplication result 5 output by the matrix multiplication operator 18 and the inverse matrix multiplication result 7 output by the matrix multiplication operator 21 are input into the point-to-point operator 15 for calculation to obtain the point-to-point calculation result 7.

[0064] The point-to-point calculation result 7 and the inverse matrix multiplication result 9 output by the matrix multiplication operator 24 are input into the point-to-point operator 16 for calculation to obtain the point-to-point calculation result 8.

[0065] The point-to-point calculation result 8 and the layer normalization result 4 output by the reverse layer normalization operator 2 are input into the point-to-point operator 17 for calculation to obtain the reverse propagation result.

[0066] Five templates are preset, namely Template 1, Template 2, Template 3, Template 4 and Template 5, among which Template 1 is a fusion operator consisting of a matrix multiplication operator, a point-to-point operator, a deformation operator and a reordering operator; Template 2 is a fusion operator consisting of a matrix multiplication operator, a point-to-point operator and a point-to-point operator; Template 3 is a fusion operator consisting of a matrix multiplication operator, a reordering operator and a deformation operator; Template 4 is a fusion operator consisting of a matrix multiplication operator and a point-to-point operator; Template 5 is a fusion operator consisting of a matrix multiplication operator, a deformation operator and a reordering operator.

[0067] Using template matching, Figure 5 The computational graph of is divided into 26 computational subgraphs (i.e., reverse subgraphs), see Figure 6 , respectively: reverse Figure 1 To reverse sub-graph 26.

[0068] In the specific implementation, since the reverse layer normalization operator 1 in the calculation graph does not match the above five templates, the reverse layer normalization operator 1 is used as the reverse normalization operator. Figure 1 Similarly, the reduction operator 1 is used as the inverse Figure 2 , using the matrix multiplication operator 9 as the inverse Figure 3 .

[0069] Since the matrix multiplication operator 10 and the point-to-point operator 12 in the computation graph match the template 4, the matrix multiplication operator 10 and the point-to-point operator 12 in the computation graph are divided into reverse sub- Figure 4 .

[0070] By analogy, other reverse subgraphs can be obtained in the same way. The specific contents of other reverse subgraphs are as follows: inverse Figure 5 Includes: reduction operator 2; inverse Figure 6 Includes: Matrix multiplication operator 11; inverse Figure 7 It includes: a matrix multiplication operator 12 and a point-to-point operator 13; inverse Figure 8 Including: reverse layer normalization operator 2; inverse Fig. 9 Includes: reduction operator 3; inverse Fig.10 Includes: Matrix multiplication operator 13; inverse Fig.11 Including: matrix multiplication operator 14, deformation operator 5, reordering operator 5; inverse Fig.12 Includes: Matrix multiplication operator 15; The reverse subgraph 13 includes: a reverse normalization operator; The reverse subgraph 14 comprises: a point-to-point operator 14; The reverse subgraph 15 includes: a matrix multiplication operator 16, a reordering operator 6, and a deformation operator 6; The reverse subgraph 16 includes: a reduction operator 4; The reverse subgraph 17 comprises: a matrix multiplication operator 17; The reverse subgraph 18 comprises: a matrix multiplication operator 18 .

[0071] The reverse subgraph 19 includes: a matrix multiplication operator 19, a reordering operator 7, and a deformation operator 7; The reverse subgraph 20 includes: a reduction operator 5; The reverse subgraph 21 comprises: a matrix multiplication operator 20; The reverse subgraph 22 includes: a matrix multiplication operator 21 and a point-to-point operator 15 .

[0072] The reverse subgraph 23 includes: a matrix multiplication operator 22, a reordering operator 8, and a deformation operator 8; The reverse subgraph 24 comprises: a reduction operator 6; The reverse subgraph 25 comprises: a matrix multiplication operator 23; The reverse subgraph 26 includes: a matrix multiplication operator 24 , a point-to-point operator 16 , and a point-to-point operator 17 .

[0073] Depend on Figure 5 and Figure 6 It can be seen that Figure 5 The computation graph shown contains 39 operators. Figure 6 After the computation graph is split, 26 computation subgraphs are obtained, that is, the number of computation subgraphs contained in the computation graph is less than the number of operators contained in the computation graph.

[0074] In some embodiments, for each computation subgraph, if the computation subgraph includes at least one operator combination for layout adjustment, the following subgraph optimization operations are performed on the computation subgraph: At least one operator combination for layout adjustment contained in the computation subgraph is replaced with an output address corresponding to each of the at least one operator combination for layout adjustment.

[0075] Specifically, the operator combination for layout adjustment includes: a deformation operator and a reordering operator. The deformation operator is executed before the reordering operator, or the deformation operator is executed after the reordering operator. This application does not specifically limit the execution order of the deformation operator and the reordering operator in the operator combination.

[0076] Since the deformation operator and the reordering operator are IO-intensive operators, their essence is to convert a data arrangement of the tensor in memory into another data arrangement in memory. Before executing the operator combination of the deformation operator and the reordering operator, the original data arrangement of the input tensor (that is, the original address in the memory) can be obtained in advance. Therefore, the output address corresponding to the target data arrangement obtained after the data arrangement of the input tensor is processed by the above operator combination can be calculated in advance. Then the output address is merged into the output of the previous operator in the operator combination in the calculation subgraph, so that the operator combination can be eliminated in the calculation subgraph. After obtaining the calculation result of the previous operator, the calculation result of the previous operator can be directly output according to the output address corresponding to the target data arrangement.

[0077] For example, see Figure 7 , assuming that in the computational graph segmentation result, the forward subgraph includes: matrix multiplication operator 5, deformation operator 4, and reordering operator 4. Perform the following optimization operations on the forward subgraph: First, the original data arrangement (i.e., original address) of the matrix multiplication result of the matrix multiplication operator 5 in the memory is obtained; then the deformation operator 4 and the reordering operator 4 are calculated to perform deformation operations and reordering operations on the original data arrangement of the matrix multiplication result in turn, and the output address corresponding to the target data arrangement is obtained. The output address is merged into the kernel function of the matrix multiplication operator 5 to eliminate the deformation operator 4 and the reordering operator 4 from the forward subgraph. In the subsequent execution of the forward subgraph, after the matrix multiplication operator 5 is executed to obtain the matrix multiplication result, the matrix multiplication result is directly output according to the output address corresponding to the target data arrangement.

[0078] For example, see Figure 8 , assuming that in the computation graph segmentation result, the reverse subgraph includes: matrix multiplication operator 16, reordering operator 6, deformation operator 6. Perform the following optimization operations on the reverse subgraph: First, the original data arrangement (i.e., original address) of the matrix multiplication result of the matrix multiplication operator 16 in the memory is obtained; then, the reordering operator 6 and the deformation operator 6 are calculated to perform reordering operations and deformation operations on the original data arrangement of the matrix multiplication result in turn, and the output address corresponding to the target data arrangement is obtained. The output address is merged into the kernel function of the matrix multiplication operator 16 to eliminate the reordering operator 6 and the deformation operator 6 from the reverse subgraph. In the subsequent execution of the reverse subgraph, after the matrix multiplication operator 16 is executed to obtain the matrix multiplication result, the matrix multiplication result is directly output according to the output address corresponding to the target data arrangement.

[0079] In an embodiment of the present application, by replacing the operator combination used for layout adjustment in the calculation subgraph with the output address corresponding to the operator combination used for layout adjustment, the delay caused by the deformation operator and the reordering operator reading and writing the video memory is reduced, thereby improving the computing performance.

[0080] Step 202, based on the resource amount of the index resources on the artificial intelligence chip, obtain the sub-graph merging granularity.

[0081] Specifically, the index resource is used to index the memory object for the input tensor or output tensor. When the index resource is the Usharp id used to index the memory object, the resource amount of the index resource refers to the number of Usharp ids. For artificial intelligence chips, the resource amount of index resources is limited; since the role of index resources is to index the memory object for the input tensor or output tensor, the total number of input tensors and output tensors contained in each operator is also limited.

[0082] Based on this, the present application determines the upper limit of the total number of input tensors and output tensors contained in a merged subgraph based on the resource amount of the index resources, and uses the upper limit as the subgraph merging granularity.

[0083] For example, if the number of Usharp IDs on the artificial intelligence chip is set to 100, the upper limit of the total number of input tensors and output tensors contained in a merged subgraph is 100.

[0084] It should be noted that the present application may first execute the subgraph optimization operation of the calculation subgraph, and then execute the operation of determining the subgraph merging granularity; it may also first execute the operation of determining the subgraph merging granularity, and then execute the subgraph optimization operation of the calculation subgraph; it may also execute the subgraph optimization operation of the calculation subgraph and the operation of determining the subgraph merging granularity in parallel; the present application does not specifically limit the execution order of the subgraph optimization operation of the calculation subgraph and the operation of determining the subgraph merging granularity.

[0085] Step 203: assemble and merge multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph.

[0086] In some embodiments, multiple computation subgraphs are assembled and merged in sequence according to the subgraph merging granularity and the execution order of the multiple computation subgraphs to obtain at least one merged subgraph.

[0087] Specifically, multiple computation subgraphs are traversed in sequence according to the execution order of the multiple computation subgraphs. In the initial stage, the first computation subgraph traversed is used as the reference subgraph. Other computation subgraphs are traversed in sequence according to the execution order of other computation subgraphs. Each time a computation subgraph is traversed, the computation subgraph is merged with the reference subgraph to obtain a merged result.

[0088] Determine whether the total number of input tensors and output tensors contained in the merged result is less than or equal to the subgraph merge granularity; if so, use the merged result as a new reference subgraph and traverse the next computational subgraph; Otherwise, the reference subgraph in the merged result is output as a merged subgraph; then the computation subgraph in the merged result is used as a new reference subgraph, and the next computation subgraph is traversed; And so on, the process stops when the traversal reaches the last computational subgraph, and the reference subgraph obtained after traversing the last computational subgraph is output as a merged subgraph.

[0089] In practical applications, when merging the reference subgraph with the computation subgraph, there may be overlapping tensors between the output tensor of the reference subgraph and the input tensor of the computation subgraph. These overlapping tensors will be converted into intermediate tensors of the merged result, rather than input tensors or output tensors of the merged result. Therefore, when counting the total number of input tensors and output tensors contained in the merged result, the total number of input tensors and output tensors contained in the computation subgraph and the total number of input tensors and output tensors contained in the reference subgraph can be summed to obtain a preliminary summation result. Then, the number of tensors converted into intermediate tensors is removed from the preliminary summation result to obtain the total number of input tensors and output tensors contained in the merged result. In other words, the total number of input tensors and output tensors contained in the merged result is less than or equal to the sum of the total number of input tensors and output tensors contained in the computation subgraph and the reference subgraph respectively.

[0090] For example, the subgraph merging granularity on the AI ​​chip is set to 9 Usharp IDs, and the computational graph is divided into 4 computational subgraphs, namely Figure 1 To calculator Figure 4 , where the computation sub Figure 1 Includes 3 input tensors and 3 output tensors; calculation sub Figure 2 Includes 1 input tensor and 1 output tensor; the calculation sub Figure 3 Includes: 6 input tensors and 3 output tensors; calculation sub Figure 4 Includes: 3 input tensors and 3 output tensors.

[0091] The calculator Figure 1 As a reference subgraph and a computation subgraph Figure 2 Merge and obtain the merge result 1. Figure 1 The total number of input tensors and output tensors included is 6, and the calculation sub Figure 2 The total number of input tensors and output tensors included is 2, and the initial sum of the two is 8. Figure 2 , that is, these two tensors are intermediate tensors of merge result 1. Therefore, the number of intermediate tensors is removed from the preliminary summation result, and the total number of input tensors and output tensors contained in merge result 1 is 6 (less than the subgraph merge granularity). Further, the reference subgraph is updated to merge result 1.

[0092] Combine the reference subgraph with the computation subgraph Figure 3 Merge and obtain the merged result 2. Since the two output tensors of the reference subgraph are Figure 3 The two input tensors of are the same, that is, these four tensors are converted into the intermediate tensors of merge result 2. Using the same method as above, the total number of input tensors and output tensors contained in merge result 2 is 11 (larger than the subgraph merging granularity). Therefore, the reference subgraph in merge result 2 (that is, merge result 1) is used as the merge subgraph. Figure 1 Output.

[0093] The calculator Figure 3 As a new reference subgraph and computation subgraph Figure 4 Merge and obtain the merged result 3. Since the three output tensors of the reference subgraph are the same as the computation subgraph Figure 4 The three input tensors of are the same, that is, these six tensors are converted into intermediate tensors of merge result 3. Using the same method as above, the total number of input tensors and output tensors contained in merge result 3 can be obtained to be 9 (equal to the subgraph merge granularity). Further, the reference subgraph is updated to merge result 3.

[0094] Because the calculator Figure 4 is the last computational subgraph, so the current reference subgraph (i.e., merged result 3) is used as the merged subgraph. Figure 2 Output.

[0095] In an embodiment of the present application, multiple computational subgraphs are assembled and merged according to the subgraph merging granularity to obtain at least one merged subgraph. This allows the computational graph to be divided into as few merged subgraphs as possible. Therefore, when each merged subgraph is compiled into a corresponding merged kernel function and executed, the number of merged kernel functions is greatly reduced, thereby reducing the time overhead of transmitting the kernel function.

[0096] In some implementations, for each merged subgraph, the following operations are performed: Each intermediate tensor of a merged subgraph is traversed, and each time an intermediate tensor is traversed, cache resources in the artificial intelligence chip are allocated to the intermediate tensor, where the intermediate tensor is a tensor transferred between computing subgraphs in a merged subgraph.

[0097] Specifically, the cache resources allocated for the intermediate tensor can be the GMB in the artificial intelligence chip.

[0098] In order to cache the entire intermediate tensor in the cache resources as much as possible to avoid fragmentation, this application prioritizes allocating cache resources to intermediate tensors that occupy less space.

[0099] Specifically, each time an intermediate tensor is traversed and it is determined that the storage space occupied by the intermediate tensor is less than a preset threshold, cache resources in the artificial intelligence chip are allocated to the intermediate tensor.

[0100] The preset threshold may be equal to the size of the cache resources on the artificial intelligence chip, or may be less than the size of the cache resources on the artificial intelligence chip.

[0101] In some embodiments, in order to effectively reuse cache resources in an artificial intelligence chip while ensuring correct calculation of operators, so as to improve the utilization of cache resources, the present application allocates cache resources to intermediate tensors based on the life cycle of each intermediate tensor.

[0102] When the life cycle of an intermediate tensor ends, the cache resources allocated to the intermediate tensor are recycled and then reallocated to other intermediate tensors to achieve cache resource reuse.

[0103] When an intermediate tensor depends on multiple computational subgraphs, the lifecycle of the intermediate tensor needs to end after all dependent computational subgraphs are executed. If there are other intermediate tensors between the computational subgraphs that the intermediate tensor depends on, in order to avoid data loss and ensure the correctness of the calculation, the cache resources allocated to the intermediate tensor do not overlap with the cache resources allocated to other intermediate tensors between the computational subgraphs that the intermediate tensor depends on. When the lifecycle of the intermediate tensor ends, the cache resources allocated to the intermediate tensor are reused.

[0104] For example, see Fig. 9 , the merged subgraph includes the computation subgraph Figure 1 , calculator Figure 2 and calculator Figure 3 , the intermediate tensors of the merged subgraphs include: Figure 1 With calculator Figure 2 The intermediate tensor F1 and the operator Figure 1 With calculator Figure 3 The intermediate tensor F1 and the operator Figure 2 With calculator Figure 3 The intermediate tensor F2 between them.

[0105] For the calculator Figure 1 With calculator Figure 2 The intermediate tensor F1 between the two is allocated the first storage space in the GMB. Figure 2 To perform calculations, you also need to enter the operator Figure 3 The calculation is performed, that is, the intermediate tensor F1 not only depends on the calculation sub Figure 2 , but also depends on the calculator Figure 3 The life cycle of the intermediate tensor F1 is in the computation sub Figure 3Therefore, the first storage space allocated for the intermediate tensor F1 cannot be allocated to the computation sub-modules that the intermediate tensor F1 depends on. Figure 2 With calculator Figure 3 The intermediate tensor F2 between the first and second storage spaces is allocated to the intermediate tensor F2. The second storage space in the GMB needs to be allocated to the intermediate tensor F2. The first storage space and the second storage space are different storage spaces in the GMB.

[0106] For example, see Fig.10 , the merged subgraph includes the computation subgraph Figure 1 , calculator Figure 2 and calculator Figure 3 , the intermediate tensors of the merged subgraphs include: Figure 1 With calculator Figure 2 The intermediate tensor F1 between the calculation sub Figure 2 With calculator Figure 3 The intermediate tensor F2 between them.

[0107] For the calculator Figure 1 With calculator Figure 2 The intermediate tensor F1 between the two is allocated the first storage space in the GMB. Since the life cycle of the intermediate tensor F1 is in the computation sub Figure 2 The process ends when the execution is completed, so the first storage space in the GMB can be reused to save the intermediate tensor F2.

[0108] Step 204: compile each merged subgraph into a corresponding merged kernel function, and execute the merged kernel function corresponding to each merged subgraph to obtain a calculation result of the encoding module.

[0109] Specifically, the central processing unit in the computer device compiles each merged subgraph into a merged kernel function (MegaKernel) and transmits the merged kernel function to the artificial intelligence chip in the computer device. The artificial intelligence chip executes the corresponding merged kernel functions in sequence according to the execution order of each merged subgraph to obtain the calculation result of the encoding module. In addition, the process of obtaining the merged subgraph described in steps 201 to 203 is also executed by the central processing unit in the computer device.

[0110] In an embodiment of the present application, the computational graph of the encoding module is divided into multiple computational subgraphs based on a preset template; then, based on the resource amount of the index resources on the artificial intelligence chip, the subgraph merging granularity is obtained; and then, according to the subgraph merging granularity, multiple computational subgraphs are assembled and merged to obtain at least one merged subgraph, that is, the computational graph is divided into as few merged subgraphs as possible while fully considering the index resources on the artificial intelligence chip. Therefore, when each merged subgraph is compiled into a corresponding merged kernel function and executed, the number of merged kernel functions is greatly reduced, thereby reducing the time overhead of transmitting the kernel function. Secondly, the computational graph optimization method of the present application not only takes into account the operators contained in the computational graph itself, but also combines the resource situation of the underlying chip that actually executes the computational graph, so that the computational graph is optimized to the maximum extent, thereby improving the optimization effect of the computational graph.

[0111] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of a computational graph optimization device, such as Fig.11 As shown, the computation graph optimization device 1100 includes: A segmentation module 1101 is used to segment the computation graph of the encoding module into multiple computation subgraphs based on a preset template; The merging module 1102 is used to obtain a subgraph merging granularity based on the resource amount of the index resources on the artificial intelligence chip; assemble and merge the multiple computing subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph; The execution module 1103 is used to compile each merged subgraph into a corresponding merged kernel function, and execute the merged kernel function corresponding to each merged subgraph to obtain the calculation result of the encoding module.

[0112] Optionally, the index resource is used to: index a video memory object for an input tensor or an output tensor; The merging module 1102 is specifically used for: Based on the resource amount of the index resource, determine an upper limit value of the total number of input tensors and output tensors included in a merged subgraph; The upper limit value is used as the sub-graph merging granularity.

[0113] Optionally, the merging module 1102 is specifically used for: According to the subgraph merging granularity and the execution order of the multiple computing subgraphs, the multiple computing subgraphs are assembled and merged in sequence to obtain at least one merged subgraph.

[0114] Optionally, the merging module 1102 is further used for: After assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph, the following operations are performed for each merged subgraph: Each intermediate tensor of a merged subgraph is traversed, and each time an intermediate tensor is traversed, cache resources in the artificial intelligence chip are allocated to the intermediate tensor, wherein the intermediate tensor is a tensor transferred between computational subgraphs in the merged subgraph.

[0115] Optionally, the merging module 1102 is specifically used for: Each time an intermediate tensor is traversed and it is determined that the storage space occupied by the intermediate tensor is less than a preset threshold, the cache resource is allocated to the intermediate tensor.

[0116] Optionally, the cache resources allocated to the intermediate tensor do not overlap with the cache resources allocated to other intermediate tensors between the computational subgraphs that the intermediate tensor depends on.

[0117] Optionally, the merging module 1102 is further used for: Before assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph, if the computation subgraph includes at least one operator combination for layout adjustment, performing the following subgraph optimization operation on the computation subgraph: The at least one operator combination for layout adjustment contained in the computation subgraph is replaced with the output addresses corresponding to the at least one operator combination for layout adjustment.

[0118] In an embodiment of the present application, the computational graph of the encoding module is divided into multiple computational subgraphs based on a preset template; then, based on the resource amount of the index resources on the artificial intelligence chip, the subgraph merging granularity is obtained; and then, according to the subgraph merging granularity, multiple computational subgraphs are assembled and merged to obtain at least one merged subgraph, that is, the computational graph is divided into as few merged subgraphs as possible while fully considering the index resources on the artificial intelligence chip. Therefore, when each merged subgraph is compiled into a corresponding merged kernel function and executed, the number of merged kernel functions is greatly reduced, thereby reducing the time overhead of transmitting the kernel function. Secondly, the computational graph optimization method of the present application not only takes into account the operators contained in the computational graph itself, but also combines the resource situation of the underlying chip that actually executes the computational graph, so that the computational graph is optimized to the maximum extent, thereby improving the optimization effect of the computational graph.

[0119] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Fig.12 As shown, it includes at least one artificial intelligence chip 100 and a memory 1201 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 1201 is not limited in the embodiment of the present application. Fig.12For example, the artificial intelligence chip 100 and the memory 1201 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0120] In an embodiment of the present application, the memory 1201 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned computational graph optimization method by executing the instructions stored in the memory 1201.

[0121] Among them, the artificial intelligence chip 100 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and the calculation graph optimization is realized by running or executing instructions stored in the memory 1201 and calling data stored in the memory 1201. Optionally, the artificial intelligence chip 100 may include one or more processing units, and the artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 1201 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.

[0122] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0123] The memory 1201 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1201 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 1201 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1201 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0124] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned computational graph optimization method.

[0125] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned computational graph optimization method.

[0126] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0131] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A computational graph optimization method, characterized in that: include: The computational graph of the encoding module is divided into multiple computational subgraphs based on a preset template; Based on the resource amount of index resources on the AI ​​chip, the subgraph merging granularity is obtained; According to the subgraph merging granularity, assembling and merging the multiple computation subgraphs to obtain at least one merged subgraph; Each merged subgraph is compiled into a corresponding merged kernel function, and the merged kernel function corresponding to each merged subgraph is executed to obtain a calculation result of the encoding module.

2. The method according to claim 1, characterized in that The index resource is used to: index a video memory object for an input tensor or an output tensor; The step of obtaining the subgraph merging granularity based on the resource amount of the index resources on the artificial intelligence chip includes: Based on the resource amount of the index resource, determine an upper limit value of the total number of input tensors and output tensors included in a merged subgraph; The upper limit value is used as the sub-graph merging granularity.

3. The method according to claim 2, characterized in that The step of assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph includes: According to the subgraph merging granularity and the execution order of the multiple computing subgraphs, the multiple computing subgraphs are assembled and merged in sequence to obtain at least one merged subgraph.

4. The method according to claim 1, characterized in that After assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph, the method further includes: For each merged subgraph, perform the following operations: Each intermediate tensor of a merged subgraph is traversed, and each time an intermediate tensor is traversed, cache resources in the artificial intelligence chip are allocated to the intermediate tensor, wherein the intermediate tensor is a tensor transferred between computational subgraphs in the merged subgraph.

5. The method according to claim 4, characterized in that Each time an intermediate tensor is traversed, a cache resource in the artificial intelligence chip is allocated to the intermediate tensor, including: Each time an intermediate tensor is traversed and it is determined that the storage space occupied by the intermediate tensor is less than a preset threshold, the cache resource is allocated to the intermediate tensor.

6. The method according to claim 4, characterized in that Also includes: The cache resources allocated for the intermediate tensor do not overlap with the cache resources allocated for other intermediate tensors between the computational subgraphs to which the intermediate tensor depends.

7. The method according to any one of claims 1 to 6, characterized in that: Before assembling and merging the multiple computation subgraphs according to the subgraph merging granularity to obtain at least one merged subgraph, the method further includes: For each computation subgraph, if the computation subgraph includes at least one operator combination for layout adjustment, the following subgraph optimization operations are performed on the computation subgraph: The at least one operator combination for layout adjustment contained in the computation subgraph is replaced with the output addresses corresponding to the at least one operator combination for layout adjustment.

8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method as claimed in any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.

Images