Method and device for determining operator fusion strategy of neural network, and storage medium

Abstract

The present disclosure relates to the technical field of computers, and comprises a method and device for determining an operator fusion strategy of a neural network and a storage medium. By determining off-chip memory parameters when a single operator is calculated in different calculation modes, first off-chip memory parameters satisfying on-chip cache constraints are obtained. For each group of operators having a fusible relationship in a plurality of operators, off-chip memory parameters when the group of operators is calculated in different calculation modes after fusion are determined, and second off-chip memory parameters satisfying on-chip cache constraints are obtained. Based on the first off-chip memory parameters of each operator in the group of operators and the second off-chip memory parameters of the group of operators, fusion benefits of the group of operators are determined. The group of operators with the largest fusion benefits is fused. The group of operators with the largest fusion benefits is found through cyclic iteration, the operator fusion strategy can be determined without designing a hardware platform, and the generality and flexibility of the method for determining the operator fusion strategy are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and particularly relates to a method and device for determining an operator fusion strategy of a neural network and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence, neural network models are also widely used. A neural network model is generally composed of multiple operators. An operator refers to a basic unit of a neural network model, such as a convolution, pooling, activation, and the like. These operators are combined in a neural network according to a certain structure to collectively complete the tasks of a neural network model, such as a feature extraction task, a data conversion task, and the like. Taking a neural network model as a large language model as an example, the large language model includes multiple layers of operators, and each operator has a parameter quantity of tens of billions and an output data result.

[0003] Due to the large amount of data of the operators in the neural network model, it is difficult to store these data in the on-chip memory of a single hardware accelerator. Therefore, at least part of the data of the operators in the neural network model needs to be stored in an off-chip storage space. At this time, during the calculation process of the neural network model, the data in the off-chip storage space needs to be read on-chip, which leads to a large number of accesses to the off-chip storage space, which affects the calculation efficiency of the neural network model.

[0004] Currently, the number of accesses to the off-chip storage space generated during the calculation process of the neural network model can be reduced through operator fusion technology. Operator fusion technology refers to merging multiple continuous independent operators into a larger operator to reduce the storage access overhead, thereby improving the calculation efficiency and performance. Operator fusion technology can reduce the number of times of writing intermediate results back to the off-chip and reading them from the off-chip again. Before operator fusion is performed, it is necessary to determine which operators to fuse, that is, to determine the operator fusion strategy of the neural network.

[0005] The conventional method for determining the operator fusion strategy of a neural network is generally based on an existing general-purpose processor (such as a CPU or a GPU) and specifically includes the following steps: classifying the operators in the neural network based on the characteristics of the general-purpose processor and evaluating the fusion potential of different operators; for each operator group with fusion potential, searching for an optimal operator combination through a search algorithm to obtain a fusion strategy; and generating code or instructions corresponding to the fusion strategy through kernel-level programming to execute the fused operators on the general-purpose processor.

[0006] However, the fusion strategy obtained based on the characteristics of the general-purpose processor may not be suitable for a special hardware architecture design, and when the hardware architecture design is not determined, the fusion strategy cannot be determined through the conventional method for determining the fusion strategy. SUMMARY

[0007] Therefore, the disclosure provides a method and device for determining an operator fusion strategy of a neural network and a storage medium, which can solve the problem that a traditional operator fusion strategy cannot determine a fusion strategy without designing a hardware platform, and improve the versatility and flexibility of the operator fusion strategy determination method.

[0008] According to an aspect of the disclosure, a method for determining an operator fusion strategy of a neural network is provided, which includes:

[0009] Obtaining tensor information of a plurality of operators in a neural network to be analyzed;

[0010] Obtaining on-chip cache constraints of a hardware platform to be designed;

[0011] For each operator, based on the tensor information of the operator, determining off-chip memory parameters when the operator is calculated by different calculation methods, to obtain first off-chip memory parameters that satisfy the on-chip cache constraints;

[0012] For each operator group having a fusion relationship in the plurality of operators, based on the tensor information of each operator in the operator group, determining off-chip memory parameters when the operator group is calculated by different calculation methods after fusion, to obtain second off-chip memory parameters that satisfy the on-chip cache constraints;

[0013] For each operator group, based on the first off-chip memory parameters of each operator in the operator group and the second off-chip memory parameters of the operator group, determining fusion benefits of the operator group;

[0014] In a case where the fusion benefits of at least one operator group meet a preset condition, performing operator fusion on an operator group having the largest fusion benefit; regarding the fused operator group as one operator, and regarding the second off-chip memory parameters corresponding to the fused operator group as the first off-chip memory parameters, triggering execution of the step of determining, for each operator group having a fusion relationship in the plurality of operators, off-chip memory parameters when the operator group is calculated by different calculation methods after fusion, based on the tensor information of each operator in the operator group, to obtain second off-chip memory parameters that satisfy the on-chip cache constraints, and the steps thereafter.

[0015] In a possible implementation, the step of determining, for each operator, off-chip memory parameters when the operator is calculated by different calculation methods, based on the tensor information of the operator, to obtain first off-chip memory parameters that satisfy the on-chip cache constraints, includes:

[0016] According to different first division parameters, performing data block division on the input data scale indicated by the tensor information, to obtain a data block scale corresponding to each first division parameter;

[0017] determine the on-chip cache space occupied when calculating according to the data block size corresponding to each first partition parameter, and a first candidate off-chip memory access parameter;

[0018] screen the first off-chip memory access parameter from each first candidate off-chip memory access parameter that satisfies the on-chip cache constraint.

[0019] In a possible implementation, the tensor information indicates that the input data size has i dimension information, and indicates that the calculation process of the operator includes loop calculation of each dimension information, where i is a positive integer; accordingly,

[0020] The determining the on-chip cache space occupied when calculating according to the data block size corresponding to each first partition parameter, and the first candidate off-chip memory access parameter includes:

[0021] determining the first candidate off-chip memory access parameter when calculating according to the data block size corresponding to each first partition parameter and different loop sequences between each dimension information.

[0022] In a possible implementation, the determining the first candidate off-chip memory access parameter when calculating according to the data block size corresponding to each first partition parameter and different loop sequences between each dimension information includes:

[0023] For each loop sequence, from the i th loop to the 1 th loop, the data block size corresponding to different first partition parameters read from the off-chip to the on-chip and the output data size obtained after each calculation process in each loop are determined layer by layer.

[0024] Based on the data block size and the output data size corresponding to each calculation, the off-chip memory access parameter corresponding to each loop is determined.

[0025] Based on the off-chip memory access parameter corresponding to each loop, the first candidate off-chip memory access parameter corresponding to each loop sequence is determined.

[0026] In a possible implementation, the determining, for each group of operators having a fusable relationship in the plurality of operators, the off-chip memory access parameter when the group of operators is calculated by different calculation manners after fusion based on the tensor information of each operator in the group of operators, to obtain the second off-chip memory access parameter satisfying the on-chip cache constraint includes:

[0027] performing data block partition on the input data size indicated by the tensor information of each operator in the group of operators according to different second partition parameters, to obtain a data block size corresponding to each second partition parameter;

[0028] determine on-chip cache space occupied when fusion computation is performed according to a data block size corresponding to each second partition parameter, and a second candidate off-chip memory access parameter;

[0029] screen the second off-chip memory access parameter from each second candidate off-chip memory access parameter that satisfies the on-chip cache constraint.

[0030] In a possible implementation, the tensor information of each operator in the operator group indicates that the input data size of the operator group has j dimension information, and indicates that the fusion computation process of the operator group includes loop computation of each dimension information, where j is a positive integer; accordingly,

[0031] The determining of the on-chip cache space occupied when fusion computation is performed according to a data block size corresponding to each second partition parameter, and the second candidate off-chip memory access parameter includes:

[0032] determining the second candidate off-chip memory access parameter when fusion computation is performed according to different loop sequences between the dimension information and the data block size corresponding to each second partition parameter.

[0033] In a possible implementation, the determining of the second candidate off-chip memory access parameter when fusion computation is performed according to different loop sequences between the dimension information and the data block size corresponding to each second partition parameter includes:

[0034] For each loop sequence, from the jth loop to the first loop, the first data block size corresponding to different second partition parameters read from the off-chip to the on-chip and the first output data size obtained after computation are determined layer by layer in each fusion computation process of each loop.

[0035] determining, based on the first output data size corresponding to each second partition parameter, the second data block size corresponding to the same second partition parameter read from the off-chip to the on-chip when the second operator in the operator group is executed, and the second output data size obtained after computation;

[0036] determining the off-chip memory access parameter corresponding to each loop based on the first data block size, the first output data size, the second data block size and the second output data size corresponding to each computation;

[0037] determining the second candidate off-chip memory access parameter corresponding to each loop sequence based on the off-chip memory access parameter corresponding to each loop.

[0038] In a possible implementation, the determining of the fusion benefit of each operator group based on the first off-chip memory access parameter of each operator in the operator group and the second off-chip memory access parameter of the operator group includes:

[0039] determining a sum of the first off-chip memory parameters of each operator in the operator group, to obtain a parameter sum;

[0040] determining a difference between the second off-chip memory parameter and the parameter sum, to obtain a fusion benefit of the operator group.

[0041] In a possible implementation, the method further includes:

[0042] In a case where the fusion benefits of each operator group do not all satisfy the preset condition, determining the design manner of the hardware platform based on the first off-chip memory parameters of each operator obtained after operator fusion.

[0043] In a possible implementation, the determining the design manner of the hardware platform based on the first off-chip memory parameters of each operator obtained after operator fusion includes:

[0044] determining a minimum off-chip memory parameter corresponding to the on-chip cache constraint based on the first off-chip memory parameters of each operator obtained after operator fusion;

[0045] determining an optimal on-chip cache constraint of the hardware platform based on a plurality of on-chip cache constraints and the minimum off-chip memory parameter corresponding to each on-chip cache constraint.

[0046] In a possible implementation, the determining the design manner of the hardware platform based on the first off-chip memory parameters of each operator obtained after operator fusion includes:

[0047] obtaining a cost function based on a relationship between the on-chip cache constraint and a hardware area of the hardware platform;

[0048] determining a minimum off-chip memory parameter corresponding to the on-chip cache constraint based on the first off-chip memory parameters of each operator obtained after operator fusion;

[0049] determining an optimal on-chip cache constraint based on a cost function value corresponding to each cache constraint in the plurality of on-chip cache constraints and the minimum off-chip memory parameter corresponding to each cache constraint.

[0050] According to another aspect of the present disclosure, a device for determining an operator fusion strategy of a neural network is provided, including a processor, a memory for storing processor-executable instructions, and wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0051] According to another aspect of the present disclosure, a non-volatile computer readable storage medium having computer program instructions stored thereon is provided, wherein the computer program instructions are executed by a processor to implement the above method.

[0052] According to another aspect of the present disclosure, there is provided a computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in a processor of an electronic device, the processor in the electronic device performs the above method.

[0053] By determining the off-chip memory access parameters of the operators when the operators are calculated in different calculation manners based on the tensor information of the single operator, the first off-chip memory access parameters satisfying the on-chip cache constraint are obtained; for each operator group having a fusable relationship in the plurality of operators, based on the tensor information of each operator in the operator group, the off-chip memory access parameters of the operator group when the operator group is fused and calculated in different calculation manners are determined, and the second off-chip memory access parameters satisfying the on-chip cache constraint are obtained; for each operator group, based on the first off-chip memory access parameters of each operator in the operator group and the second off-chip memory access parameters of the operator group, the fusion benefit of the operator group is determined; in the case that the fusion benefit of at least one operator group meets a preset condition, the operator group with the largest fusion benefit is fused; the operator group with the largest fusion benefit is found through cyclic iteration and lookup, and the operator fusion strategy satisfying the on-chip cache constraint can be determined; the operator fusion strategy can be determined without designing a hardware platform; the problem that the traditional operator fusion strategy cannot determine the fusion strategy without designing a hardware platform can be solved, and the versatility and flexibility of the operator fusion strategy determination method are improved.

[0054] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0056] Figure 1 A flowchart showing a method for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure is shown;

[0057] Figure 2 A schematic diagram showing the types of operators according to an embodiment of the present disclosure is shown;

[0058] Figure 3 A schematic diagram showing the operation logic of a matrix multiplication operator according to an embodiment of the present disclosure is shown;

[0059] Figure 4 A schematic diagram showing the block operation logic of a matrix multiplication operator according to an embodiment of the present disclosure is shown;

[0060] Figure 5A schematic diagram showing a data flow relationship between various operators in a neural network according to an embodiment of the present disclosure;

[0061] Figure 6 A schematic diagram showing a fusable type combination of operators according to an embodiment of the present disclosure;

[0062] Figure 7 A schematic diagram showing operation logic of a element-wise addition operator according to an embodiment of the present disclosure;

[0063] Figure 8 A schematic diagram showing fused computation logic of a matrix multiplication operator and an element-wise addition operator according to an embodiment of the present disclosure;

[0064] Figure 9 A block diagram of a device for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure;

[0065] Figure 10 A block diagram of a device for determining an operator fusion strategy of a neural network according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] Various exemplary embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0067] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0068] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the present disclosure can be omitted. It will be appreciated that the present disclosure can be practiced with the exact details as

[0069] Figure 1 A flowchart of a method for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure is shown. The present embodiment takes the method as an example for use in an electronic device with computing capability, which can be a user terminal or a server. The user terminal can be a computer, a tablet, a mobile phone, etc. The present embodiment does not limit the implementation of the electronic device. As shown in the figure, the method comprises: Figure 1

[0070] In step 101, tensor information of a plurality of operators in a neural network to be analyzed is acquired.​

[0071] In this application, a neural network includes a plurality of operators, which, as mentioned above, refer to basic units constituting a neural network model. Generally, the calculation process of an operator includes calculating input data according to the calculation logic of the operator to obtain output data.

[0072] Different types of operators correspond to different calculation logics. Referring to Figure 2 , the types of operators in a neural network include but are not limited to:

[0073] Matrix and matrix operations, such as: matrix multiplication (MM) operators, convolution (conv) operators, sparse MM operators;

[0074] Matrix and vector operations or vector and vector operations, such as: matrix-vector multiplication (MV) operators, normalization Norm operators (such as root mean square normalization RMS Norm, layer normalization Layer Norm, group normalization Group Norm, etc.), Softmax operator, etc.

[0075] Scalar operations, such as: element-wise addition operators, element-wise multiplication operators, activation function (such as Relu, Gelu, Silu, etc.) operators, basic functions (such as sine function sin, cosine function cos, natural exponential function exp, square root function sqrt, etc.);

[0076] Reordering or reshaping operations, such as: transpose functions, concat functions, reshape functions, padding functions, interpolate functions, etc.

[0077] In the calculation process of each operator, the input data and the output data exist in the form of tensors. The input data scale of each operator and the output data scale obtained based on the input data scale are usually fixed. The on-chip cache of the hardware occupied in the calculation process of the operator is related to the input data scale and the output data scale. Based on this, in this embodiment, the tensor information of each operator is used to indicate the input data scale of the operator. In addition, in order to determine the output data scale based on the input data scale, the tensor information of the operator can also include the type of the operator, so as to determine the calculation logic of the operator according to the type, and then determine the output data scale based on the input data scale according to the calculation logic.

[0078] The data scale (including the input data scale and the output data scale) refers to the size and dimension of the data set.

[0079] For example, the operator is an MM operator, the input data of the operator is two matrices A and B, the input data scale corresponding to the matrix A is [batch_size, M, K], and the input data scale corresponding to the matrix B is [K, N]. At this time, the input data scale corresponding to the matrix A is three dimensions, the first dimension batch_size indicates the number of matrix A in a batch of data, the second dimension M indicates that the matrix A has M rows, and the third dimension K indicates that the matrix A has K columns. The input data scale corresponding to the matrix B is two dimensions, the first dimension K indicates that the matrix B has K rows, and the second dimension N indicates that the matrix B has N columns. In other embodiments, other types of operators can also be represented by similar tensor information, which will not be enumerated one by one in this embodiment.

[0080] The network information of the neural network to be analyzed is stored in the electronic device, and the network information includes the tensor information of a plurality of operators. The tensor information of each operator can be obtained by reading and analyzing the network information.

[0081] In step 102, the on-chip cache constraint of the hardware platform to be designed is obtained.

[0082] In this embodiment, the hardware platform to be designed refers to a hardware platform that has not been designed yet, and the on-chip cache constraint is used to indicate the maximum value of the cache that can be directly integrated into the processor core of the hardware platform.

[0083] Optionally, the on-chip cache constraint can be received through a human-computer interaction interface, or can be sent by other devices, and the embodiment does not limit the acquisition method of the on-chip cache constraint.

[0084] Optionally, when designing the hardware platform, the on-chip cache constraint can be designed according to the on-chip cache constraint, or can be designed according to the on-chip cache constraint. In this embodiment, the on-chip cache constraint is used to guide the design of the hardware platform, or the hardware platform can refer to the on-chip cache constraint during design. The way the on-chip cache constraint guides the design of the hardware platform is referred to in the following embodiments, which will not be described here.

[0085] In addition, step 102 can be executed after step 101; or can be executed before step 101; or can be executed synchronously with step 101, and the embodiment does not limit the execution order between steps 101 and 102.

[0086] In step 103, for each operator, based on the tensor information of the operator, off-chip memory access parameters of the operator when the operator is calculated in different calculation manners are determined, and first off-chip memory access parameters satisfying the on-chip cache constraint are obtained.

[0087] Off-chip memory access refers to an operation of a processor core when accessing data in a memory outside the processor core, such as an operation of accessing data in a dynamic random access memory (DRAM). Since off-chip memory access involves data transmission between different physical locations, off-chip memory access is much slower than on-chip access. An off-chip memory access parameter is used to indicate the amount of data of off-chip memory access. The off-chip memory access parameter can be represented by an off-chip memory access amount, and in other embodiments, the off-chip memory access parameter can also be other parameters indicating the amount of data of off-chip memory access. The present embodiment does not limit the implementation manner of the off-chip memory access parameter.

[0088] The data scale of the operator is usually large. In order to ensure that the first off-chip memory access parameter of a single operator satisfies the on-chip cache constraint, the data scale of the operator needs to be block-divided, so as to reduce the data scale of the operator. At this time, different calculation manners include different block division manners, and the first division parameters corresponding to different block division manners are different. In the present embodiment, different types of operators have different operation logics, and each operation logic can have different block division manners.

[0089] Correspondingly, for each operator, based on the tensor information of the operator, off-chip memory access parameters of the operator when the operator is calculated in different calculation manners are determined, and first off-chip memory access parameters satisfying the on-chip cache constraint are obtained, including steps 1031-1033:

[0090] In step 1031, the input data scale indicated by the tensor information is block-divided according to different first division parameters, and a data block scale corresponding to each first division parameter is obtained.

[0091] The data block scale obtained by dividing according to the first division parameter is less than or equal to the input data scale.

[0092] In one example, the first partition parameter sequentially traverses values in a preset range with a preset step size, and a maximum value of the preset range is less than or equal to the input data scale indicated by the tensor information. For example, the tensor information is: matrix, shape [batch_size, M, K]. Here, matrix indicates that the type of the operator is a matrix-matrix operation, and shape indicates the input data scale. Accordingly, the input data scale indicated by the tensor information is [batch_size, M, K]. Illustratively, the first partition parameter includes three dimensions corresponding to preset ranges, the preset range corresponding to the first dimension is [1, batch_size], the preset range corresponding to the second dimension is [1, M], and the preset range corresponding to the third dimension is [1, N]. The preset step size can be 1 or other values greater than 1, and the embodiment does not limit the value of the preset step size.

[0093] Optionally, the traversal value can be sequentially taken from large to small in the preset range with a preset step size, and after the first partition parameter is obtained each time, steps 1032 and 1033 are executed. At this time, if the first partition parameter obtained from the maximum value in the preset range can meet the on-chip cache constraint, it is not necessary to continue to obtain the first partition parameter from the preset range, and the calculation resources consumed in the process of determining the operator fusion strategy can be saved. In other embodiments, the traversal value can be sequentially taken from small to large with a preset step size, and / or after a plurality of first partition parameters are obtained, steps 1032 and 1033 are executed to obtain the first candidate off-chip memory parameter corresponding to each first partition parameter. The embodiment does not limit the traversal value.

[0094] In another example, the first partition parameter is searched according to an exponential search algorithm. Illustratively, the values of the preset range are arranged from large to small, and the index 2 0 indicates the first partition parameter. After steps 1032-1033 are executed, the index 2 0 corresponding to the first candidate off-chip memory parameter is obtained; the index 2 i indicates the first partition parameter. After steps 1032-1033 are executed, the index 2 i corresponding to the first candidate off-chip memory parameter is obtained; if the index 2 i corresponding to the first candidate off-chip memory parameter decreases, the value of i is updated, i is a value that gradually increases from 1; if the index 2 i corresponding to the first candidate off-chip memory parameter increases, the preset range is reduced to the index [2 i-1 , 2 i-1] indicates, in the narrowed range, each first partition parameter according to a search algorithm to perform steps 1032-1033.

[0095] In this example, the search space of the first partition parameter is narrowed by the exponential search algorithm, so that the electronic device does not need to traverse all the first partition parameters in the preset range, and the determination efficiency of the first off-chip memory parameter is improved.

[0096] In other embodiments, the electronic device can also search for the first partition parameter based on other search algorithms to improve the search efficiency of the first partition parameter, such as searching based on binary search, etc., and the implementation of the search algorithm is not limited in this embodiment.

[0097] Step 1032, determine the on-chip cache space occupied when calculated according to the data block size corresponding to each first partition parameter, and the first candidate off-chip memory parameter.

[0098] After partitioning the input data size of the operator, the operation logic of the operator is not affected, so the electronic device determines the on-chip cache space corresponding to different data block sizes in the calculation process based on the operation logic of the operator.

[0099] For example, referring to the operation logic of the MM operator shown in Figure 3 It can be known from Figure 3 that the input data of the operator is matrix A and matrix B, the input data size shape of matrix A is [batch_size, M, K], the input data size shape corresponding to matrix B is [K, N], and the input data size shape of matrix C is [batch_size, M, N]. The operation logic of the MM operator includes: for each batch index value b from [0, batch_size-1], calculating the product of the value of the m+1th row and the k+1th column of the A+1th batch of matrix A and the value of the k+1th row and the n+1th column of matrix B, and determining the value of the m+1th row and the n+1th column of the C+1th batch of matrix C based on each product. Wherein, m is an index value from [0, M-1] in turn; n is an index value from [0, N-1] in turn; k is an index value from [0, K-1] in turn.

[0100] Based on this operation logic, if all the data of the MM operator is stored in the off-chip storage space, such as in the off-chip DRAM, the data needs to be read from the off-chip to the on-chip buffer buffer during the calculation process. Referring to Figure 4If the first partitioning parameter is (sub_batch, sub_m, sub_n, sub_k), where sub_batch is used to partition batch_size, sub_m is used to partition M, sub_n is used to partition N, and sub_k is used to partition K, then the data block size corresponding to matrix A is... The size of the data block corresponding to matrix B is: Accordingly, by performing matrix multiplication based on the data block size corresponding to matrix A and the data block size corresponding to matrix B, the data block size corresponding to matrix C is obtained as follows: in, This indicates rounding x up. According to... Figure 3 As can be seen from the operational logic shown, Figure 4 In the above, for each data block obtained according to the first partitioning parameter, the calculation process for the data block corresponding to matrix C is as follows:

[0101] C[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1,n ite* sub_n: +sub_n-1]+=A[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1,k ite* sub_k: +sub_k-1]*B[k] ite* sub_k: +sub_k-1,n ite* sub_n: +sub_n-1];

[0102] Where x:+y represents the interval [x, x+y], and x+=y represents x=x+y. That is, the above formula means that for the b-th matrix A... ite +1 batch block (m) ite +1)×(k ite +1) matrix blocks, the batch index range [b ite *sub_batch, b ite The batch and row index range [m] corresponding to each batch index in [sub_batch+sub_batch-1] is... ite* sub_m, m ite* The value corresponding to each row index in sub_m+sub_m-1] and the column index range [k] ite* sub_k, k ite* The value corresponding to each column index in [sub_k+sub_k-1] is related to the (k)th column index of matrix B.ite +1)×(n ite In +1) matrix blocks, the row index range is [k ite* sub_k, k ite* The value corresponding to each row index in sub_k+sub_k-1], and the column index range [m ite* sub_m, m ite* Multiply the values ​​corresponding to the column indices of [sub_m+sub_m-1], and determine the b-th column in matrix C based on the products. ite +1 batch block (m) ite +1)×(n ite +1) values ​​of matrix blocks.

[0103] according to Figure 4 It can be seen that b ite From The index value of the value, m ite From The index of the value, n ite From The index value of the value, k ite From The index of the value.

[0104] Then, based on the calculation logic of the data blocks and different caching mechanisms, the on-chip cache space occupied when calculating the data block size corresponding to each first partitioning parameter is determined.

[0105] In one example, if the on-chip caching mechanism is a ping-pong cache, there are two buffers (usually called buffer A and buffer B). These two buffers take turns storing input and output data. That is, when one buffer is used to store input data, the other buffer is used to store output data. According to the principle of the ping-pong cache, both buffers in the on-chip cache need to store the data block corresponding to the input data, and one buffer needs to store the data block corresponding to the output data. Therefore, in... Figure 4 In the data block size shown, the on-chip cache space occupied by each data block in each calculation process is: 2*sub_batch*sub_m*sub_k+2*sub_k*sub_n+sub_batch*sub_m*sub_n.

[0106] In another example, if the on-chip caching mechanism is a single cache, meaning there is only one buffer on the chip, then this buffer needs to store the data blocks corresponding to the input data and the data blocks corresponding to the output data. Therefore, in Figure 4In the illustrated data block size, the on-chip cache space occupied by each data block in each calculation process is: sub_batch*sub_m*sub_k+sub_k*sub_n+sub_batch*sub_m*sub_n.

[0107] In other embodiments, based on different on-chip cache mechanisms, the cache space occupied by each data block in each calculation process can also be other values, which will not be described here in this embodiment.

[0108] Optionally, in addition to the influence of the data block size on the off-chip memory parameters, the loop order (i.e., data flow) of reading data blocks from off-chip to on-chip can also affect the off-chip memory parameters. At this time, different calculation methods also include calculation according to different loop orders. The tensor information indicates that the input data size has i dimension information, and indicates that the calculation process of the operator includes loop calculation of each dimension information, and i is a positive integer. Accordingly,

[0109] Determining the on-chip cache space occupied when calculating according to the data block size corresponding to each first partition parameter, and the first candidate off-chip memory parameter, includes: determining the first candidate off-chip memory parameter when calculating according to the different loop orders between the dimension information and the data block size corresponding to each first partition parameter.

[0110] In one example, the electronic device can determine the first candidate off-chip memory parameter corresponding to different first partition parameters for each loop order. At this time, determining the first candidate off-chip memory parameter when calculating according to the different loop orders between the dimension information and the data block size corresponding to each first partition parameter includes:

[0111] For each loop order, from the i-th layer loop to the 1st layer loop, the data block size corresponding to different first partition parameters read from off-chip to on-chip in each calculation process of each layer loop and the output data size obtained after each calculation are determined layer by layer; based on the data block size and the output data size corresponding to each calculation, the off-chip memory parameter corresponding to each layer loop is determined; the first candidate off-chip memory parameter corresponding to each loop order is determined based on the off-chip memory parameter corresponding to each layer loop. At this time, the first candidate off-chip memory parameter includes the off-chip memory parameter corresponding to each loop order.

[0112] Wherein, a layer loop refers to a process of traversing each value of the dimension information corresponding to the layer loop and performing calculation based on each value; each calculation refers to a process of reading the data block size according to the current value of the dimension information corresponding to each layer loop, and performing calculation based on the operation logic of the operator. Accordingly, since the dimension information corresponding to each layer loop has multiple values, each layer loop has multiple calculations.

[0113] That is, if there are i layers of loops, first iterate reversely from the i-th layer of loop, the i-th layer of loop performs the data block to be read in each time of on-chip calculation, at this time, the data block size of the input data and the data block size of the output data corresponding to different first division parameters can be obtained. Then, the overall off-chip memory access parameter corresponding to the data block iterated and calculated by the i-th layer of loop is determined, and the tensor data access parameter not affected is unchanged. Then, jump to the i-1-th layer of loop, continue to determine the overall off-chip memory access parameter corresponding to the data block iterated and calculated by the i-1-th layer of loop, until the iteration of each layer is completed.

[0114] For example: Figure 4 The tensor has 4 layers of loops, that is, i = 4, and the first layer of loop to the fourth layer of loop are b ite The loop in which m ite The loop in which n ite The loop in which k ite The loop in which k ite The loop in which k ite Each value of k ite corresponds to a calculation, and the calculation of the fourth layer of loop is completed by traversing all values of k ite When the loop in which k ite is iterated, only two input matrices exist in the k dimension and need to be iterated, and after the iteration, the data block size of the input matrix becomes [sub_batch, sub_m, K] and [K, sub_m, sub_n], and the data block size of the output matrix becomes [sub_batch, sub_m, sub_n]. For the loop in which n Figure 4 The off-chip memory access parameter Memory access corresponding to the loop sequence shown in the formula (1) (that is, i = 4, and the first candidate off-chip memory access parameter of the loop sequence 4-1) is represented by the following formula:

[0115]

[0116] If i = 1, that is, the coefficient outside each bracket in the Memory access calculation formula is 1, the first candidate off-chip memory access parameter corresponding to the case without loop can be obtained.

[0117] In another example, the electronic device can also determine, for each first partition parameter, a first candidate off-chip memory parameter corresponding to a different loop order, and the calculation principle is the same as above. This embodiment does not limit the determination manner of the first candidate off-chip memory parameter.

[0118] In step 1033, the first off-chip memory parameter is screened from the first candidate off-chip memory parameters that satisfy the on-chip cache constraint.

[0119] According to the above, satisfying the on-chip cache constraint means that the on-chip cache space occupied by the calculation process is less than the on-chip cache constraint Buffer_size.

[0120] For example, the on-chip cache space corresponding to the above ping-pong cache: sub_batch*sub_m*sub_k+sub_k*sub_n+sub_batch*sub_m*sub_n< Buffer_size, that is, satisfying the on-chip cache constraint.

[0121] For another example, the on-chip cache space corresponding to the above single cache: sub_batch*sub_m*sub_k+sub_k*sub_n+sub_batch*sub_m*sub_n< Buffer_size, that is, satisfying the on-chip cache constraint.

[0122] In one example, screening the first off-chip memory parameter from the first candidate off-chip memory parameters that satisfy the on-chip cache constraint includes: determining the smallest first candidate off-chip memory parameter from the first candidate off-chip memory parameters that satisfy the on-chip cache constraint, to obtain the first off-chip memory parameter.

[0123] In other embodiments, the manner of screening the first off-chip memory parameter can also be other manners, and this embodiment does not limit the screening manner of the first off-chip memory parameter.

[0124] In step 104, for each group of operators with a fusable relationship in the plurality of operators, based on the tensor information of each operator in the group of operators, the off-chip memory parameter when the group of operators is fused and calculated by different calculation manners is determined, to obtain a second off-chip memory parameter that satisfies the on-chip cache constraint.

[0125] In one example, the group of operators with a fusable relationship means that the same group of output results of a first operator are input to at least one second operator, at this time, the first operator and the second operator have a fusable relationship, and the second operator is also called a successor operator of the first operator. It is assumed that the data flow relationship between each operator in the neural network is as shown in the following table. Figure 5 Figure 5 ​Each circle represents an operator, and the connection between the operators represents the activation value that needs to be transmitted through the off-chip storage. Figure 5 It can be known that the output result 1 of the operator A is input to the operator B and the operator C, and the output result 2 of the operator A is input to the operator D, at this time, the operator A and the operators B and C have a fusible relationship, that is, can be taken as a group of operators; the operator A and the operator D have a fusible relationship, that is, can be taken as a group of operators.

[0126] In another example, the group of operators with a fusible relationship refers to that the same group of output results of a first operator is input to at least one second operator, and the type of the first operator and the type of the second operator match a preset fusible type combination.

[0127] For example: the probability of fusion between operators with a specific type is relatively large, therefore, a fusible type combination can be constructed based on the specific type, the type combination composed of the type of the first operator and the type of the second operator is matched with the fusible type combination, if the matching is successful, the first operator and the second operator are taken as an operator group; if the matching is unsuccessful, the first operator and the second operator are not taken as an operator group. At this time, for different operators with a relatively small fusion probability, the off-chip memory parameters after fusion are not needed, and the computing resources of the electronic device can be saved.

[0128] The matching success refers to that the type of the first operator and the type of the second operator are the same as each type in the fusible type combination. For example: referring to each fusible type combination shown in Figure 6 The first operator and the second operator matched with each fusible type combination include:

[0129] The type combination of vector + element, and the first operator and the second operator matched with the combination include: MV operator + activation function operator or residual module operator;

[0130] The type combination of vector + vector, and the first operator and the second operator matched with the combination include: MV operator + bidirectional Softmax operator, or MV operator + bidirectional Norm operator, or MV operator + MV operator (wherein, each MV operator can also be inserted with an element-wise Element-wise operation);

[0131] The type combination of matrix + element, and the first operator and the second operator matched with the combination include: MM operator or conv operator + activation function operator, or MM operator or conv operator + residual add operator;

[0132] The type combination of matrix + vector, and the first operator and the second operator matched with the combination include: MM operator + bidirectional Softmax operator, or MM operator + bidirectional RSM Norm operator, or LayerNorm operator + conv operator;

[0133] The type combination of matrix + matrix, the first operator and the second operator matched with the combination include: MM operator + MM operator, or batch MM operator + Softmax operator (the Softmax operator can be a conversion on the input of vector level or element level) + batch MM operator, or conv operator + MM operator, or conv operator + conv operator, or MM operator + conv operator, or MM operator + MM operator + LayerNorm operator.

[0134] The type combination of parallel matrix, the first operator and the second operator matched with the combination include: Query, Key, Value (QKV) vector, the QKV vector can be an MM base architecture based on the MM operator or a conv base architecture based on the conv operator, or a Feed-Forward Networks (FFN) operator, the FFN operator can be an FFN layer in a large language model Llama.

[0135] The type combination of matrix or vector + rearrangement, the first operator and the second operator matched with the combination include: padding function + conv operator, or interpolate function + conv operator, or MM operator + transpose function, or MV operator + concat function, or MM operator + reshape function.

[0136] Figure 6 The illustrated fusible type combinations are only illustrative, and in actual implementation, the fusible type combinations can include more or fewer types, which are not enumerated one by one in this embodiment.

[0137] Optionally, the manner in which the electronic device obtains each group of operator groups having a fusible relationship includes: converting a data flow relationship between operators in a neural network into a graph network, wherein the operators are nodes in the graph network, and the data flow relationship between the operators is an edge in the graph network; the node features include but are not limited to: node type and tensor information of the node; the edge features include but are not limited to: data flow relationship between nodes; and the graph network is parsed to obtain each group of operator groups having a fusible relationship.

[0138] In other embodiments, the neural network can not be converted into a graph network, but the network information of the neural network is directly parsed to obtain the operator groups having a fusible relationship, and the embodiment does not limit the manner in which the operator groups having a fusible relationship are obtained.

[0139] In this embodiment, fused computation refers to a computation method where the output data of the first operator in the operator group is not written back to off-chip storage but is retained in the on-chip cache space for use by the second operator. Therefore, the on-chip cache space usage during operator fused computation is larger than that of a single operator. To ensure that the second off-chip memory access parameters meet on-chip cache constraints during fused computation, the data size of the operator group needs to be partitioned into blocks to reduce the data size of the operator group. At this time, the computation method includes block partitioning methods, and different block partitioning methods correspond to different second partitioning parameters.

[0140] Accordingly, for each group of operators that have a fusion relationship among multiple operators, based on the tensor information of each operator in the operator group, the off-chip memory access parameters when the operator group is fused and calculated using different calculation methods are determined, and the second off-chip memory access parameters that satisfy the on-chip cache constraints are obtained, including steps 1041-1043:

[0141] Step 1041: According to different second partitioning parameters, the input data scale indicated by the tensor information of each operator in the operator group is divided into data blocks to obtain the data block scale corresponding to each second partitioning parameter.

[0142] The method for obtaining the second partitioning parameter is described in the above description of the first partitioning parameter, and will not be repeated here in this embodiment.

[0143] Step 1042: Determine the on-chip cache space occupied when performing fusion calculations according to the data block size corresponding to each second partitioning parameter, and the second candidate off-chip memory access parameters.

[0144] Based on the calculation logic of the data block and different caching mechanisms, determine the on-chip cache space occupied when calculating the data block size corresponding to each second partitioning parameter.

[0145] Compared to the on-chip cache space corresponding to a single operator, the on-chip cache space occupied during fusion computation needs to be increased by the data block size corresponding to the output data of the first operator.

[0146] For example: the first operator in the operator group is the MM operator, and the second operator is the element-wise add operator. The operational logic of the element-wise add operator is referenced below. Figure 7 As shown, according to Figure 7It can be seen that the input data of this operator are matrices A and B. The input data size shape of matrix A is [batch_size, M, N], the corresponding input data size shape of matrix B is [batch_size, M, N], and the input data size shape of matrix C is [batch_size, M, N]. The operation logic of the elementwise add operator includes: for each batch index value b from [0, batch_size-1], calculate the sum of the value of the (m+1)th row and (n+1)th column of matrix A and the value of the (m+1)th row and (n+1)th column of matrix B to obtain the value of the (m+1)th row and (n+1)th column of matrix C. Here, m is the index value taken sequentially from [0, M-1]; n is the index value taken sequentially from [0, N-1].

[0147] Combination Figure 4 The operational logic of the MM operator shown is as follows: Figure 7 The operational logic of the elementwise add operator shown can be used to obtain the operational logic of the MM operator + elementwise add operator fusion calculation. (Refer to...) Figure 8 As shown, the first operator performs matrix multiplication on input data blocks A and B to obtain D, which is then used by the second operator. The second operator performs element-wise addition on D and input data block C to obtain the updated D.

[0148] That is, for each data block obtained according to the second partitioning parameter, the calculation process for the data block corresponding to matrix D is as follows:

[0149] D[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1,n ite* sub_n: +sub_n-1]+=A[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1,k ite* sub_k: +sub_k-1]*B[k] ite* sub_k: +sub_k-1,n ite* sub_n: +sub_n-1];

[0150] D[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1,n ite* sub_n: +sub_n-1]+=C[b itesub_batch + sub_batch - 1, m ite* sub_m + sub_m - 1, n ite* sub_n + sub_n - 1.

[0151] where x + y denotes the interval [x, x + y] and x + = y denotes x = x + y. That is, the above formula denotes that, for the b ite + 1) x (k ite + 1) matrix blocks corresponding to the matrix A, the value of each batch index in the batch index range [b ite * sub_batch, b ite * sub_batch + sub_batch - 1] corresponds to a batch, the value of each row index in the row index range [m ite sub_m, m ite* sub_m + sub_m - 1] corresponds to a value, the value of each column index in the column index range [k ite* sub_k, k ite* sub_k + sub_k - 1] corresponds to a value, and the value of each row index in the row index range [k ite* sub_k, k ite sub_k + sub_k - 1] corresponds to a value, the value of each column index in the column index range [m ite sub_m, m ite* sub_m + sub_m - 1] corresponds to a value, the value of each row index in the row index range [k ite* sub_k, k ite* sub_k + sub_k - 1] corresponds to a value, and the value of each column index in the column index range [m ite* sub_m, m ite sub_m + sub_m - 1] corresponds to a value, the value of each row index in the row index range [k ite sub_k, k ite sub_k + sub_k - 1] corresponds to a value, and the value of each column index in the column index range [m ite sub_m, m ite sub_m + sub_m - 1] corresponds to a value, the value of each row index in the row index range [k ite sub_k, k ite sub_k + sub_k - 1] corresponds to a value, and the value of each column index in the column index range [m ite sub_m, m ite sub_m + sub_m - 1] corresponds to a value, the value of each row index in the row index range [k ite sub_k, k ite sub_k + sub_k - 1] corresponds to a value, and the value of each column index in the column index range [m ite sub_m, m

[0152] Then, the value of the b ite + 1) x (n ite + 1) matrix blocks in the D matrix is added element by element with the value of the b ite + 1) x (n ite + 1) matrix blocks in the matrix C to obtain the value of the b ite + 1) x (n ite + 1) matrix blocks in the updated D matrix. ite sub_m, m

[0153] According to Figure 8It can be seen that b ite is an index value taken from m ite is an index value taken from n ite is an index value taken from k ite is an index value taken from .

[0154] Correspondingly, if the on-chip cache mechanism is the ping-pong cache, the on-chip cache space occupied by each data block in each calculation process is: 2*sub_batch*sub_m*sub_k+2*sub_k*sub_n+2sub_batch*sub_m*sub_n. Compared with the on-chip cache space corresponding to the single MM operator shown in FIG. 8, the data block size sub_batch*sub_m*sub_n of the output data block D of the first operator is increased. Figure 4

[0155] If the on-chip cache mechanism is the single cache, the on-chip cache space occupied by each data block in each calculation process is: sub_batch*sub_m*sub_k+sub_k*sub_n+2sub_batch*sub_m*sub_n. Similarly, compared with the on-chip cache space corresponding to the single MM operator shown in FIG. 9, the data block size sub_batch*sub_m*sub_n of the output data block D of the first operator is increased. Figure 4

[0156] According to the foregoing, the loop order (that is, the data flow) of the data block read from the off-chip to the on-chip in the fusion calculation process can also affect the off-chip memory access parameter. Based on this, the tensor information of each operator in the operator group indicates the input data size of the operator group and has j dimension information, and the fusion calculation process of the operator group includes loop calculation of each dimension information, and j is a positive integer. Correspondingly, the on-chip cache space occupied when fusion calculation is performed according to the data block size corresponding to each second division parameter, and the second candidate off-chip memory access parameter, include: determining the second candidate off-chip memory access parameter when fusion calculation is performed according to the different loop orders between each dimension information and the data block size corresponding to each second division parameter.

[0157] In one example, the second candidate off-chip memory access parameter when fusion calculation is performed according to the different loop orders between each dimension information and the data block size corresponding to each second division parameter includes:

[0158] ​​For each loop sequence, from the j-th loop to the 1-th loop, the size of the first data block corresponding to the different second partitioning parameters read from outside the chip and the size of the first output data obtained after the calculation are determined layer by layer during each fusion calculation process of each loop.

[0159] Determine the first output data size corresponding to each second partitioning parameter, and when executing the second operator in the operator group, read the second data block size corresponding to the same second partitioning parameter from outside the chip to the chip, as well as the second output data size obtained after calculation.

[0160] Based on the first data block size, first output data size, second data block size and second output data size corresponding to each calculation, determine the off-chip memory access parameters corresponding to each loop level.

[0161] The second candidate external memory access parameter for each loop sequence is determined based on the external memory access parameters corresponding to each loop level.

[0162] for example: Figure 8 The loop has 4 nested loops, i.e., i=4. The 4th nested loop of the 1st nested loop is as follows: b ite The loop in which m ite The loop in which n is located ite The loop in which it is located, and k ite The loop in which it is located. At this point, from the fourth level k... ite The loop begins by reading in input matrices of sizes [sub_batch, sub_m, sub_k] and [sub_k, sub_n] corresponding to the first operator from outside the chip, and output matrices of sizes [sub_batch, sub_m, sub_n]. Then, it reads in input matrices of sizes [sub_batch, sub_m, sub_n] corresponding to the second operator from outside the chip, and determines the output matrix of sizes [sub_batch, sub_m, sub_n] based on the first and second output data sizes. In summary, Figure 8 The off-chip memory access parameter Memoryaccess corresponding to the loop sequence shown is expressed by the following formula:

[0163]

[0164] As can be seen from the above, compared with the single MM operator, the off-chip memory access parameters corresponding to the loop order in the fusion calculation process become larger.

[0165] Step 1043: Select second off-chip memory access parameters from the second candidate off-chip memory access parameters that satisfy the on-chip cache constraints.

[0166] The manner of screening the second off-chip memory parameter is the same as that of screening the first off-chip memory parameter, and the related description is detailed in the manner of screening the first off-chip memory parameter above. The present embodiment will not be described again here.

[0167] In step 105, for each group of operator groups, the fusion benefit of the operator group is determined based on the first off-chip memory parameters of each operator in the operator group and the second off-chip memory parameter of the operator group.

[0168] In one example, for each group of operator groups, the fusion benefit of the operator group is determined based on the first off-chip memory parameters of each operator in the operator group and the second off-chip memory parameter of the operator group, including: determining the sum of the first off-chip memory parameters of each operator in the operator group to obtain a parameter sum; determining the difference between the second off-chip memory parameter and the parameter sum to obtain the fusion benefit of the operator group.

[0169] In other embodiments, the quotient between the second off-chip memory parameter and the parameter sum can also be determined to obtain the fusion benefit of the operator group. The present embodiment does not limit the manner of determining the fusion benefit.

[0170] In step 106, in the case where the fusion benefit of at least one group of operator groups meets a preset condition, the operator group with the largest fusion benefit is fused; the fused operator group is taken as one operator, and the second off-chip memory parameter corresponding to the fused operator group is taken as the first off-chip memory parameter, triggering the execution of the step of determining the off-chip memory parameter when the operator group fused based on the tensor information of each operator in the operator group is calculated by different calculation manners to obtain the second off-chip memory parameter meeting the on-chip cache constraint, and the steps thereafter, i.e., the step 104.

[0171] In one example, if the fusion benefit is the difference between the second off-chip memory parameter and the parameter sum, correspondingly, the fusion benefit meeting the preset condition can be that the fusion benefit is greater than 0; in other embodiments, the fusion benefit can also be greater than a preset value, and the preset value is greater than 0. The present embodiment does not limit the value of the preset value.

[0172] In another example, if the fusion benefit is the quotient between the second off-chip memory parameter and the parameter sum, correspondingly, the fusion benefit meeting the preset condition can be that the fusion benefit is greater than 1, and in other embodiments, the fusion benefit can also be greater than a preset quotient value, and the preset quotient value is greater than 1. The present embodiment does not limit the value of the preset quotient value.

[0173] In summary, the method for determining the operator fusion strategy of the neural network provided in the embodiment determines the off-chip memory parameters when the operators are calculated in different calculation manners based on the tensor information of a single operator, obtains the first off-chip memory parameters that meet the on-chip cache constraint; for each operator group having a fusion relationship in the plurality of operators, determines the off-chip memory parameters when the operator group is calculated in different calculation manners based on the tensor information of each operator in the operator group, and obtains the second off-chip memory parameters that meet the on-chip cache constraint; for each operator group, determines the fusion benefit of the operator group based on the first off-chip memory parameters of each operator in the operator group and the second off-chip memory parameters of the operator group; in the case that the fusion benefit of at least one operator group meets a preset condition, the operator group with the largest fusion benefit is fused; the operator group with the largest fusion benefit is found through cyclic iteration, and the operator fusion strategy that meets the on-chip cache constraint can be determined; the operator fusion strategy can be determined without designing a hardware platform, and the problem that the traditional operator fusion strategy cannot determine the fusion strategy without designing a hardware platform can be solved, and the versatility and flexibility of the method for determining the operator fusion strategy are improved.

[0174] Optionally, based on the above embodiment, in the case that the fusion benefits of the various operator groups do not meet the preset condition, the design manner of the hardware platform is determined based on the first off-chip memory parameters of each operator obtained after the operator fusion.

[0175] According to the above step 106, after the operator group is fused, the fused operator group is treated as an operator, and therefore, each operator obtained after the operator fusion includes each operator that does not participate in the fusion and the operator corresponding to the entire fused operator group.

[0176] In one example, determining the design manner of the hardware platform based on the first off-chip memory parameters of each operator obtained after the operator fusion includes:

[0177] Based on the first off-chip memory parameters of each operator obtained after the operator fusion, the minimum off-chip memory parameter corresponding to the on-chip cache constraint is determined; and based on the plurality of on-chip cache constraints and the minimum off-chip memory parameter corresponding to each on-chip cache constraint, the optimal on-chip cache constraint of the hardware platform is determined.

[0178] Specifically, a sum of the first off-chip memory parameters of the respective operators is calculated to obtain a minimum off-chip memory parameter; the on-chip cache constraint is gradually increased, and the minimum off-chip memory parameter corresponding to each on-chip cache constraint is determined according to the above embodiment. Generally, as the on-chip cache constraint increases, the corresponding minimum off-chip memory parameter decreases, but when the on-chip cache constraint increases to a certain extent, the minimum off-chip memory parameter will not continue to decrease, or the decrease will not be too large, and then, further increasing the on-chip cache constraint will not have a great impact on the minimum off-chip memory parameter. Based on this, based on the various on-chip cache constraints and the minimum off-chip memory parameter corresponding to each on-chip cache constraint, the optimal on-chip cache constraint of the hardware platform is determined, including:

[0179] For the on-chip cache constraints sorted from small to large and the minimum off-chip memory parameter corresponding to each on-chip cache constraint, it is determined whether the difference between the minimum off-chip memory parameters corresponding to two adjacent on-chip cache constraints is less than or equal to a preset difference range; if the difference is less than or equal to the preset difference range, the smaller on-chip cache constraint of the two adjacent on-chip cache constraints is determined as the optimal on-chip cache constraint.

[0180] Or,

[0181] Based on the various on-chip cache constraints and the minimum off-chip memory parameter corresponding to each on-chip cache constraint, a relationship curve between the on-chip cache constraint and the minimum off-chip memory parameter is generated; a curve point on the relationship curve whose absolute value of the slope is less than a preset absolute value is determined, and the on-chip cache constraint corresponding to the curve point is determined as the optimal on-chip cache constraint.

[0182] The determination manner of the optimal on-chip cache constraint can also be other manners, which will not be described here in detail.

[0183] In another example, based on the first off-chip memory parameter of each operator obtained after the operator fusion, the design manner of the hardware platform is determined, including:

[0184] The cost function based on the relationship between the on-chip cache constraint and the hardware area of the hardware platform is obtained; the minimum off-chip memory parameter corresponding to the on-chip cache constraint is determined based on the first off-chip memory parameter of each operator obtained after the operator fusion; and the optimal on-chip cache constraint is determined based on the cost function value corresponding to each cache constraint in the various on-chip cache constraints and the minimum off-chip memory parameter corresponding to each cache constraint.

[0185] Since the hardware area of the hardware platform is related to the manufacturing cost of the hardware, by obtaining the cost function and determining the optimal on-chip cache constraint based on the cost function value, the manufacturing cost of the hardware platform can also be saved while ensuring a small off-chip memory parameter.

[0186] Optionally, the cost function can be sent by other devices, or can also be fitted based on the correspondence between each on-chip cache constraint and hardware area, and the embodiment does not limit the acquisition manner of the cost function.

[0187] Based on the cost function value corresponding to each cache constraint in the plurality of on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each cache constraint, the optimal on-chip cache constraint is determined, including:

[0188] The cost function value corresponding to each on-chip cache constraint and the minimum off-chip memory access parameter are input into a preset mathematical model, and the on-chip cache constraint corresponding to the maximum of the output result of the data model is determined as the optimal on-chip cache constraint.

[0189] The data model is represented by the following formula:

[0190] 1 / (MemoryAccess*Cost)

[0191] Wherein, MemoryAccess represents the minimum off-chip memory access parameter, and Cost represents the cost function value.

[0192] In other embodiments, the mathematical model can also be MemoryAccess*Cost, and at this time, the on-chip cache constraint corresponding to the maximum of the output result of the data model is determined as the optimal on-chip cache constraint, and the embodiment does not limit the implementation manner of the mathematical model and the determination manner of the optimal on-chip cache constraint.

[0193] In the embodiment, the optimal on-chip cache constraint is determined through the relationship between the on-chip cache constraint and the minimum off-chip memory access parameter, which can determine the optimal on-chip cache constraint of the hardware platform before the hardware platform is designed, so that the cache space is not wasted, the off-chip memory access parameter is guaranteed to be small, and the manufacturing cost of the hardware platform is saved.

[0194] Figure 9 A block diagram of a device for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure is shown, and the device includes an information acquisition module 910, a constraint acquisition module 920, a first determination module 930, a second determination module 940, a benefit determination module 950, and an operator fusion module 960.

[0195] The information acquisition module 910 is configured to acquire tensor information of a plurality of operators in a neural network to be analyzed.

[0196] The constraint acquisition module 920 is configured to acquire an on-chip cache constraint of a hardware platform to be designed.

[0197] The first determination module 930 is configured to determine, for each operator, off-chip memory access parameters when the operator is calculated in different calculation manners based on tensor information of the operator, to obtain first off-chip memory access parameters that satisfy the on-chip cache constraint.

[0198] The second determination module 940 is configured to determine, for each operator group having a fusable relationship in the plurality of operators, off-chip memory access parameters when the operator group is fused and calculated in different calculation manners based on tensor information of each operator in the operator group, to obtain second off-chip memory access parameters that satisfy the on-chip cache constraint.

[0199] The benefit determination module 950 is configured to determine, for each operator group, a fusion benefit of the operator group based on the first off-chip memory access parameters of each operator in the operator group and the second off-chip memory access parameters of the operator group.

[0200] The operator fusion module 960 is configured to, in a case where the fusion benefit of at least one operator group satisfies a preset condition, fuse an operator group having the largest fusion benefit; take the fused operator group as one operator, and take the second off-chip memory access parameters corresponding to the fused operator group as the first off-chip memory access parameters, to trigger execution of the step of determining, for each operator group having a fusable relationship in the plurality of operators, off-chip memory access parameters when the operator group is fused and calculated in different calculation manners based on tensor information of each operator in the operator group, to obtain second off-chip memory access parameters that satisfy the on-chip cache constraint, and steps subsequent to the step.

[0201] The related descriptions are referred to the above method embodiments.

[0202] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For brevity, details are not described here.

[0203] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0204] The embodiments of the present disclosure also provide an electronic device, including a processor, and a memory for storing processor-executable instructions, wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0205] The embodiments of the present disclosure further provide a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device performs the above method.

[0206] Figure 10 is a block diagram of a determination apparatus 1900 of an operator fusion strategy of a neural network according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a server or a terminal device. Referring to Figure 10 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0207] The apparatus 1900 can also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0208] In exemplary embodiments, a non-volatile computer readable storage medium, such as the memory 1932 including computer program instructions executable by the processing component 1922 of the apparatus 1900 to complete the above method is also provided.

[0209] The above has described the embodiments of the present disclosure, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical applications or technical improvements in the art of the embodiments, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining an operator fusion strategy for a neural network, characterized in that, The method includes: Obtain tensor information of multiple operators in the neural network to be analyzed; Obtain the on-chip cache constraints of the hardware platform to be designed; For each operator, based on the tensor information of the operator, the off-chip memory access parameters when the operator is calculated according to different calculation methods are determined, and the first off-chip memory access parameters that satisfy the on-chip cache constraints are obtained. For each group of operators that can be fused among the multiple operators, based on the tensor information of each operator in the operator group, the off-chip memory access parameters when the operator group is fused and calculated using different calculation methods are determined, and the second off-chip memory access parameters that satisfy the on-chip cache constraints are obtained. For each operator group, the fusion benefit of the operator group is determined based on the first external memory access parameters of each operator in the operator group and the second external memory access parameters of the operator group. If the fusion benefit of at least one group of operators meets the preset conditions, the group of operators with the largest fusion benefit is fused; the fused group of operators is taken as an operator and the second external memory access parameter corresponding to the fused group of operators is taken as the first external memory access parameter. The following steps are triggered: for each group of operators with fusion relationship among the multiple operators, the external memory access parameters when the fused group of operators is calculated by different calculation methods based on the tensor information of each operator in the group of operators are determined, and the second external memory access parameter that satisfies the on-chip cache constraint is obtained.

2. The method according to claim 1, characterized in that, For each operator, based on the tensor information of the operator, the off-chip memory access parameters when the operator performs calculations according to different calculation methods are determined, resulting in the first off-chip memory access parameters that satisfy the on-chip cache constraints, including: According to different first partitioning parameters, the input data scale indicated by the tensor information is divided into data blocks to obtain the data block scale corresponding to each first partitioning parameter; Determine the on-chip cache space occupied when calculating the data block size corresponding to each first partitioning parameter, and the first candidate off-chip memory access parameters; The first off-chip memory access parameters are selected from the first candidate off-chip memory access parameters that satisfy the on-chip cache constraints.

3. The method according to claim 2, characterized in that, The tensor information indicates that the input data scale has i dimensions, and indicates that the calculation process of the operator includes iterative calculation of each dimension, where i is a positive integer; Accordingly, The determination of the on-chip cache space occupied when calculating the data block size according to each first partitioning parameter, and the first candidate off-chip memory access parameters, includes: Determine the first candidate off-chip memory access parameter when calculating based on different cyclic orders of information across various dimensions and the data block size corresponding to each first partitioning parameter.

4. The method according to claim 3, characterized in that, The determination of the first candidate off-chip memory access parameters when calculating based on different cyclical orders among the information of each dimension and the data block size corresponding to each first partitioning parameter includes: For each loop sequence, from the i-th loop to the 1-th loop, the size of the data block corresponding to the different first partitioning parameters read from outside the chip to the chip and the size of the output data obtained after each calculation are determined layer by layer during each calculation process of each loop. Based on the data block size and output data size corresponding to each calculation, determine the off-chip memory access parameters corresponding to each loop level; The first candidate external memory access parameter for each loop sequence is determined based on the external memory access parameters corresponding to each loop level.

5. The method according to claim 1, characterized in that, For each group of operators with fusion relationships among the plurality of operators, based on the tensor information of each operator in the group, the off-chip memory access parameters when the fused group of operators is calculated using different computation methods are determined, resulting in the second off-chip memory access parameters that satisfy the on-chip cache constraints, including: According to different second partitioning parameters, the input data scale indicated by the tensor information of each operator in the operator group is divided into data blocks to obtain the data block scale corresponding to each second partitioning parameter. Determine the on-chip cache space occupied when performing fusion calculations according to the data block size corresponding to each second partitioning parameter, and the second candidate off-chip memory access parameters; The second off-chip memory access parameters are selected from the second candidate off-chip memory access parameters that satisfy the on-chip cache constraints.

6. The method according to claim 5, characterized in that, The tensor information of each operator in the operator group indicates that the input data scale of the operator group has j dimensions, and indicates that the fusion calculation process of the operator group includes iterative calculation of each dimension, where j is a positive integer; correspondingly, The determination of the on-chip cache space occupied during the fusion calculation according to the data block size corresponding to each second partitioning parameter, and the second candidate off-chip memory access parameters, includes: Determine the second candidate off-chip memory access parameters when performing fusion calculations based on different cyclical orders between information of each dimension and the data block size corresponding to each second partitioning parameter.

7. The method according to claim 6, characterized in that, The determination of the second candidate off-chip memory access parameters when performing fusion calculations based on different cyclical orders among the information of each dimension and the data block size corresponding to each second partitioning parameter includes: For each loop sequence, from the j-th loop to the 1-th loop, the size of the first data block corresponding to the different second partitioning parameters read from outside the chip and the size of the first output data obtained after calculation are determined layer by layer during each fusion calculation process of each loop. Determine the first output data size corresponding to each second partitioning parameter, and when executing the second operator in the operator group, read the second data block size corresponding to the same second partitioning parameter from outside the chip to on the chip, and the second output data size obtained after calculation. Based on the first data block size, first output data size, second data block size and second output data size corresponding to each calculation, determine the off-chip memory access parameters corresponding to each loop level. The second candidate external memory access parameter for each loop sequence is determined based on the external memory access parameters corresponding to each loop level.

8. The method according to any one of claims 1 to 7, characterized in that, For each operator group, the fusion benefit of the operator group is determined based on the first external memory access parameters of each operator in the operator group and the second external memory access parameters of the operator group, including: Determine the sum of the first external memory access parameters of each operator in the operator group to obtain the parameter sum; The difference between the second external memory access parameter and the sum of the parameters is determined to obtain the fusion benefit of the operator group.

9. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If the fusion benefits of each operator group do not meet the preset conditions, the design method of the hardware platform is determined based on the first external memory access parameters of each operator obtained after operator fusion.

10. The method according to claim 9, characterized in that, The design method of the hardware platform is determined based on the first external memory access parameters of each operator obtained after operator fusion, including: Based on the first off-chip memory access parameters of each operator obtained after operator fusion, the minimum off-chip memory access parameter corresponding to the on-chip cache constraint is determined. The optimal on-chip cache constraint for the hardware platform is determined based on a variety of on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each on-chip cache constraint.

11. The method according to claim 9, characterized in that, The design method of the hardware platform is determined based on the first external memory access parameters of each operator obtained after operator fusion, including: Obtain a cost function, which is based on the relationship between the on-chip cache constraint and the hardware area of ​​the hardware platform; Based on the first off-chip memory access parameters of each operator obtained after operator fusion, the minimum off-chip memory access parameter corresponding to the on-chip cache constraint is determined. The optimal on-chip cache constraint is determined based on the cost function value corresponding to each cache constraint among various on-chip cache constraints, and the minimum off-chip memory access parameter corresponding to each cache constraint.

12. A device for determining an operator fusion strategy for a neural network, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 11 when executing instructions stored in the memory.

13. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 11.

Images