Method and device for determining operator fusion strategy of neural network and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a method and a device for determining an operator fusion strategy of a neural network, and a storage medium. The method comprises the following steps: determining off-chip memory access parameters when a single operator performs calculation according to different calculation modes to obtain first off-chip memory access parameters meeting on-chip cache constraints; for each operator group with a fusible relationship in the plurality of operators, determining off-chip memory access parameters when calculation is performed through different calculation modes after the operator groups are fused, and obtaining second off-chip memory access parameters meeting on-chip cache constraints; based on the first off-chip memory access parameter of each operator in the operator group and the second off-chip memory access parameter of the operator group, the fusion income of the operator group is determined; carrying out operator fusion on the operator group with the maximum fusion income; the operator group with the maximum fusion income is searched through circular traversal, and an operator fusion strategy can be determined under the condition that a hardware platform is not designed; and the universality and the flexibility of the method for determining the operator fusion strategy are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method, device and storage medium for determining an operator fusion strategy of a neural network. Background Art

[0002] With the development of artificial intelligence, neural network models are also widely used. Neural network models are generally composed of multiple operators. Operators refer to the basic units that constitute neural network models, such as convolution, pooling, activation and other operators. These operators are combined in a certain structure in the neural network to complete the tasks of the neural network model, such as feature extraction tasks, data conversion tasks, etc. Taking the neural network model as a large language model as an example, the large language model contains multiple layers of operators, each of which has billions of parameters and output data results.

[0003] Since the amount of data of operators in the neural network model is large, it is difficult to store this data in the on-chip memory of a single hardware accelerator. Therefore, it is necessary to store at least part of the data of operators in the neural network model in the off-chip storage space. At this time, during the calculation process of the neural network model, the data in the off-chip storage space needs to be read on-chip, resulting in a large amount of off-chip storage space access, which will affect the calculation efficiency of the neural network model.

[0004] Currently, the number of accesses to off-chip storage space generated during the calculation of the neural network model can be reduced through operator fusion technology. Operator fusion technology refers to the technology of merging multiple consecutive independent operators into a larger operator to reduce the overhead of storage access, thereby improving computing efficiency and performance. Operator fusion technology can reduce the number of times intermediate results are written back to the chip and read from the chip again. Before performing operator fusion, it is necessary to determine in advance which operators to fuse, that is, to determine the operator fusion strategy of the neural network.

[0005] The method for determining the operator fusion strategy of traditional neural networks is generally based on existing general-purpose processors (such as CPUs or GPUs), and specifically includes: classifying the operators in the neural network based on the characteristics of the general-purpose processor and evaluating the fusion potential of different operators; for each operator group with fusion potential, searching for the optimal operator combination through a search algorithm to obtain a fusion strategy; generating the code or instructions corresponding to the fusion strategy through kernel-level programming to execute the fused operators on the general-purpose processor.

[0006] However, the fusion strategy obtained based on the characteristics of a general-purpose processor may not be applicable to a dedicated hardware architecture design, and when the hardware architecture design is not determined, the fusion strategy cannot be determined by a traditional fusion strategy determination method. Summary of the invention

[0007] In view of this, the present disclosure proposes a method, device and storage medium for determining the operator fusion strategy of a neural network, which can solve the problem that the traditional operator fusion strategy cannot determine the fusion strategy without designing a hardware platform, and improve the versatility and flexibility of the operator fusion strategy determination method.

[0008] According to one aspect of the present disclosure, a method for determining an operator fusion strategy of a neural network is provided, the method comprising:

[0009] Obtain tensor information of multiple operators in the neural network to be analyzed;

[0010] Obtain on-chip cache constraints of the hardware platform to be designed;

[0011] For each operator, based on the tensor information of the operator, determine the off-chip memory access parameter of the operator when performing calculations in different calculation modes, and obtain a first off-chip memory access parameter that satisfies the on-chip cache constraint;

[0012] For each group of operators in the plurality of operators that have a fusionable relationship, based on tensor information of each operator in the operator group, determining off-chip memory access parameters when the operator group is fused and calculated by different calculation methods, to obtain a second off-chip memory access parameter that satisfies the on-chip cache constraint;

[0013] For each operator group, determining a fusion benefit of the operator group based on a first off-chip memory access parameter of each operator in the operator group and a second off-chip memory access parameter of the operator group;

[0014] When the fusion benefit of at least one group of operator groups meets the preset conditions, the operator group with the largest fusion benefit is subjected to operator fusion; the fused operator group is taken as one operator, and the second off-chip memory access parameter corresponding to the fused operator group is taken as the first off-chip memory access parameter, triggering the execution of the step of determining, for each group of operator groups having a fusible relationship among the multiple operators, the off-chip memory access parameter when the operator group is calculated by different computing methods after fusion based on the tensor information of each operator in the operator group, and obtaining the second off-chip memory access parameter that meets the on-chip cache constraint and subsequent steps.

[0015] In a possible implementation, for each operator, based on the tensor information of the operator, determining the off-chip memory access parameter of the operator when calculating according to different calculation modes, and obtaining the first off-chip memory access parameter that satisfies the on-chip cache constraint includes:

[0016] According to different first division parameters, the input data size indicated by the tensor information is divided into data blocks to obtain a data block size corresponding to each first division parameter;

[0017] Determine the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter, and the first candidate off-chip memory access parameter;

[0018] The first off-chip memory access parameter is selected from each first candidate off-chip memory access parameter whose on-chip cache space satisfies the on-chip cache constraint.

[0019] In a possible implementation, the tensor information indicates that the input data scale has i-dimensional information, and indicates that the calculation process of the operator includes a cyclic calculation of each dimensional information, where i is a positive integer; accordingly,

[0020] The determining of the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter and the first candidate off-chip memory access parameter includes:

[0021] Determine the first candidate off-chip memory access parameter when calculating according to different loop orders between the dimensional information and the data block size corresponding to each first partition parameter.

[0022] In a possible implementation, determining the first candidate off-chip memory access parameter when calculating according to different loop orders between the dimensional information and the data block size corresponding to each first partition parameter includes:

[0023] For each loop sequence, from the i-th loop to the 1-th loop, determine layer by layer the data block size corresponding to the different first partitioning parameters read from the outside to the chip during each calculation process of each loop, and the output data size obtained after each calculation is completed;

[0024] Based on the data block size and output data size corresponding to each calculation, determine the off-chip memory access parameters corresponding to each layer of loop;

[0025] A first candidate off-chip memory access parameter corresponding to each loop sequence is determined based on the off-chip memory access parameters corresponding to each layer of loop.

[0026] In a possible implementation, for each group of operators in the multiple operators that have a fusionable relationship, determining, based on tensor information of each operator in the operator group, off-chip memory access parameters when the operator group is fused and calculated by different calculation methods, to obtain a second off-chip memory access parameter that satisfies the on-chip cache constraint includes:

[0027] According to different second partitioning parameters, the input data scale indicated by the tensor information of each operator in the operator group is partitioned into data blocks to obtain a data block scale corresponding to each second partitioning parameter;

[0028] Determine the on-chip cache space occupied when performing fusion calculation according to the data block size corresponding to each second partition parameter, and the second candidate off-chip memory access parameter;

[0029] The second off-chip memory access parameter is screened from each second candidate off-chip memory access parameter whose on-chip cache space satisfies the on-chip cache constraint.

[0030] In a possible implementation, the tensor information of each operator in the operator group indicates that the input data scale of the operator group has j-dimensional information, and indicates that the fusion calculation process of the operator group includes a cyclic calculation of each dimensional information, and j is a positive integer; accordingly,

[0031] The determining of the on-chip cache space occupied when performing fusion calculation according to the data block size corresponding to each second partition parameter and the second candidate off-chip memory access parameter includes:

[0032] Determine the second candidate off-chip memory access parameter when performing fusion calculation according to different loop orders between the dimensional information and the data block size corresponding to each second partition parameter.

[0033] In a possible implementation, determining the second candidate off-chip memory access parameter when performing fusion calculation according to different loop orders between the dimensional information and the data block size corresponding to each second partition parameter includes:

[0034] For each loop sequence, from the j-th loop to the 1-th loop, determine layer by layer, in each fusion calculation process of each loop, the first data block size corresponding to the different second partitioning parameters read from the outside to the chip when executing the first operator in the operator group, and the first output data size obtained after the calculation;

[0035] Determine the first output data scale corresponding to each second partition parameter, the second data block scale corresponding to the same second partition parameter read from off-chip to on-chip when executing the second operator in the operator group, and the second output data scale obtained after calculation;

[0036] Determine the off-chip memory access parameter corresponding to each layer of loop based on the first data block size, the first output data size, the second data block size, and the second output data size corresponding to each calculation;

[0037] The second candidate off-chip memory access parameter corresponding to each loop sequence is determined based on the off-chip memory access parameter corresponding to each layer of loop.

[0038] In a possible implementation, for each operator group, determining the fusion benefit of the operator group based on a first off-chip memory access parameter of each operator in the operator group and a second off-chip memory access parameter of the operator group includes:

[0039] Determine the sum of the first off-chip memory access parameters of each operator in the operator group to obtain a parameter sum;

[0040] The difference between the second off-chip memory access parameter and the sum of the parameters is determined to obtain a fusion benefit of the operator group.

[0041] In a possible implementation, the method further includes:

[0042] When the fusion benefits of each operator group do not meet the preset conditions, the design mode of the hardware platform is determined based on the first off-chip memory access parameters of each operator obtained after the operator fusion.

[0043] In a possible implementation, determining the design mode of the hardware platform based on the first off-chip memory access parameter of each operator obtained after the operator fusion includes:

[0044] Determine the minimum off-chip memory access parameter corresponding to the on-chip cache constraint based on the first off-chip memory access parameter of each operator obtained after the operator fusion;

[0045] Based on multiple on-chip cache constraints and minimum off-chip memory access parameters corresponding to each on-chip cache constraint, an optimal on-chip cache constraint of the hardware platform is determined.

[0046] In a possible implementation, determining the design mode of the hardware platform based on the first off-chip memory access parameter of each operator obtained after the operator fusion includes:

[0047] Obtaining a cost function, the cost function being based on a relationship between the on-chip cache constraint and a hardware area of ​​the hardware platform;

[0048] Determine the minimum off-chip memory access parameter corresponding to the on-chip cache constraint based on the first off-chip memory access parameter of each operator obtained after the operator fusion;

[0049] An optimal on-chip cache constraint is determined based on a cost function value corresponding to each cache constraint among the multiple on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each cache constraint.

[0050] According to another aspect of the present disclosure, there is provided a device for determining an operator fusion strategy of a neural network, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0051] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0052] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0053] Based on the tensor information of a single operator, the off-chip memory access parameters of the operator when calculating in different calculation methods are determined, and the first off-chip memory access parameters that meet the on-chip cache constraint are obtained; for each group of operator groups with a fusionable relationship among multiple operators, based on the tensor information of each operator in the operator group, the off-chip memory access parameters when the operator group is calculated in different calculation methods after fusion are determined, and the second off-chip memory access parameters that meet the on-chip cache constraint are obtained; for each group of operator groups, the fusion benefit of the operator group is determined based on the first off-chip memory access parameters of each operator in the operator group and the second off-chip memory access parameters of the operator group; when the fusion benefit of at least one group of operator groups meets the preset conditions, the operator group with the largest fusion benefit is fused; by looping through to find the operator group with the largest fusion benefit, an operator fusion strategy that meets the on-chip cache constraint can be determined; the operator fusion strategy can be determined without designing a hardware platform; the problem that the traditional operator fusion strategy cannot determine the fusion strategy without designing the hardware platform can be solved, and the versatility and flexibility of the operator fusion strategy determination method can be improved.

[0054] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0056] Figure 1 A flowchart showing a method for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure;

[0057] Figure 2 A schematic diagram showing the types of operators according to an embodiment of the present disclosure;

[0058] Figure 3 A schematic diagram describing the operation logic of a matrix multiplication operator according to an embodiment of the present disclosure is shown;

[0059] Figure 4 A schematic diagram illustrating the block operation logic of a matrix multiplication operator according to an embodiment of the present disclosure is shown;

[0060] Figure 5A schematic diagram showing the data flow relationship between various operators in a neural network according to an embodiment of the present disclosure;

[0061] Figure 6 A schematic diagram showing a combination of fusible types of operators according to an embodiment of the present disclosure;

[0062] Figure 7 A schematic diagram illustrating the operation logic of an element-by-element addition operator according to an embodiment of the present disclosure is shown;

[0063] Figure 8 A schematic diagram describing the fusion calculation logic of a matrix multiplication operator and an element-by-element addition operator according to an embodiment of the present disclosure is shown;

[0064] Fig. 9 A block diagram showing a device for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure;

[0065] Fig.10 A block diagram of a device for determining an operator fusion strategy of a neural network according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0066] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0067] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0068] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0069] Figure 1 A flowchart of a method for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure is shown. This embodiment is described by taking the method used in an electronic device with computing capabilities as an example. The electronic device may be a user terminal or a server. The user terminal may be a computer, a tablet computer, a mobile phone, etc. This embodiment does not limit the implementation of the electronic device. Figure 1 As shown, the method includes:

[0070] Step 101, obtaining tensor information of multiple operators in the neural network to be analyzed.

[0071] In the present application, a neural network includes a plurality of operators. As mentioned above, an operator refers to a basic unit constituting a neural network model. Generally, the calculation process of an operator includes: calculating input data according to the calculation logic of the operator to obtain output data.

[0072] Different types of operators have different corresponding calculation logics. Figure 2 , the types of operators in neural networks include but are not limited to:

[0073] Matrix-matrix operations, such as matrix multiplication (MM), convolution (conv), and sparse MM operators;

[0074] Matrix-vector operations or vector-vector operations, such as matrix-vector multiplication (MV) operator, normalization Norm operator (such as RMS Norm, layer Norm, group Norm, etc.), normalized exponential function (Softmax) operator, etc.;

[0075] Scalar operations, such as element-wise addition operator, element-wise multiplication operator, activation function (such as Relu, Gelu, Silu, etc.) operator, basic functions (such as sine function sin, cosine function cos, natural exponential function exp, square root function sqrt, etc.);

[0076] Rearrange or reshape operations, such as: transpose function, concat function, reshape function, padding function, interpolate function, etc.

[0077] During the calculation process of each operator, the input data and output data exist in the form of tensors. The input data scale of each operator and the output data scale obtained based on the input data scale are usually fixed. The on-chip cache of the hardware occupied during the operator calculation process is related to the input data scale and the output data scale. Based on this, in this embodiment, the tensor information of each operator is used to indicate the input data scale of the operator. In addition, in order to determine the output data scale based on the input data scale, the tensor information of the operator can also include the type of the operator, so as to determine the calculation logic of the operator according to the type, so as to determine the output data scale based on the input data scale according to the calculation logic.

[0078] Among them, data scale (including input data scale and output data scale) refers to the size and dimension of the data set.

[0079] For example: the operator is an MM operator, and the input data of the operator are two matrices A and B. The input data scale corresponding to matrix A is [batch_size, M, K], and the input data scale corresponding to matrix B is [K, N]. At this time, the input data scale corresponding to matrix A is three dimensions. The first dimension batch_size represents the number of matrices A in a batch of data, the second dimension M represents that matrix A has M rows, and the third dimension K represents that matrix A has K columns. The input data scale corresponding to matrix B is two dimensions. The first dimension K represents that matrix B has K rows, and the second dimension N represents that matrix B has N columns. In other embodiments, other types of operators can also be represented by similar tensor information, which will not be listed one by one in this embodiment.

[0080] The electronic device stores network information of a neural network to be analyzed, and the network information includes tensor information of multiple operators. The tensor information of each operator can be obtained by reading and parsing the network information.

[0081] Step 102: Obtain on-chip cache constraints of the hardware platform to be designed.

[0082] In this embodiment, the hardware platform to be designed refers to a hardware platform that has not yet been designed, and the on-chip cache constraint is used to indicate the maximum value of the cache that can be directly integrated into the processor core of the hardware platform.

[0083] Optionally, the on-chip cache constraint may be received through a human-computer interaction interface, or may be sent by other devices. This embodiment does not limit the method for obtaining the on-chip cache constraint.

[0084] Optionally, when actually designing the hardware platform, the on-chip cache constraint may or may not be designed according to the on-chip cache constraint. In this embodiment, the on-chip cache constraint is used to guide the design of the hardware platform, or the on-chip cache constraint can be referred to when designing the hardware platform. The way in which the on-chip cache constraint guides the design of the hardware platform is referred to the embodiment below, and this embodiment will not be elaborated here.

[0085] In addition, step 102 may be executed after step 101; or, may be executed before step 101; or, may be executed synchronously with step 101. This embodiment does not limit the execution order between steps 101 and 102.

[0086] Step 103 : For each operator, based on the tensor information of the operator, determine the off-chip memory access parameters when the operator is calculated according to different calculation methods, and obtain the first off-chip memory access parameters that meet the on-chip cache constraint.

[0087] Off-Chip Memory Access refers to the operation when the processor core accesses data in the memory located outside the processor core, such as the operation when accessing data in Dynamic Random Access Memory (DRAM). Since off-chip memory access involves the transmission of data between different physical locations, off-chip memory access is much slower than on-chip access. The off-chip memory access parameter is used to indicate the amount of data accessed off-chip. The off-chip memory access parameter can be represented by the off-chip memory access amount. In other embodiments, the off-chip memory access parameter can also be other parameters indicating the amount of data accessed off-chip. This embodiment does not limit the implementation method of the off-chip memory access parameter.

[0088] The data scale of an operator is usually large. In order to ensure that the first off-chip memory access parameter of a single operator satisfies the on-chip cache constraint, the data scale of the operator needs to be divided into blocks, thereby reducing the data scale of the operator. At this time, different calculation methods include different block division methods, and different block division methods correspond to different first division parameters. In this embodiment, different types of operators have different operation logics, and each operation logic can have different block division methods.

[0089] Accordingly, for each operator, based on the tensor information of the operator, the off-chip memory access parameters when the operator is calculated according to different calculation methods are determined to obtain the first off-chip memory access parameters that meet the on-chip cache constraint, including steps 1031-1033:

[0090] Step 1031, dividing the input data size indicated by the tensor information into data blocks according to different first division parameters, and obtaining the data block size corresponding to each first division parameter.

[0091] The size of the data block obtained by dividing according to the first division parameter is smaller than or equal to the size of the input data.

[0092] In one example, the first partition parameter takes values ​​in sequence within a preset range according to a preset step size, and the maximum value of the preset range is less than or equal to the input data scale indicated by the tensor information. For example: the tensor information is: matrix, shape [batch_size, M, K]. Wherein, matrix indicates that the type of operator is an operation between matrices, and shape indicates the input data scale. Correspondingly, the input data scale indicated by the tensor information is [batch_size, M, K]. Schematically, the first partition parameter includes preset ranges corresponding to three dimensions, respectively, the preset range corresponding to the first dimension is [1, batch_size], the preset range corresponding to the second dimension is [1, M], and the preset range corresponding to the third dimension is [1, N]. The preset step size can be 1, or it can also be other values ​​greater than 1. This embodiment does not limit the value of the preset step size.

[0093] Optionally, the method of traversing and obtaining values ​​may be to obtain values ​​in sequence from large to small according to a preset step size within a preset range, and after each value is obtained by obtaining the first partition parameter, step 1032 and step 1033 are executed. At this time, if the first partition parameter obtained by obtaining the maximum value within the preset range can satisfy the on-chip cache constraint, there is no need to continue to obtain the first partition parameter from the preset range, which can save the computing resources consumed in the process of determining the operator fusion strategy. In other embodiments, the method of traversing and obtaining values ​​may also be to obtain values ​​in sequence from small to large according to a preset step size, and / or, after obtaining multiple first partition parameters, steps 1032 and 1033 are executed to obtain the first candidate off-chip memory access parameters corresponding to each first partition parameter. This embodiment does not limit the method of traversing and obtaining values.

[0094] In another example, the first partition parameter is searched according to an exponential search algorithm. Schematically, the values ​​in the preset range are arranged from large to small, and the index 2 is searched in the preset range. 0 The first partition parameter indicated, after executing steps 1032-1033, obtains index 2 0 Corresponding first candidate off-chip memory access parameter; search index 2 within the preset range i The first partition parameter indicated, after executing steps 1032-1033, obtains index 2 i The corresponding first candidate off-chip memory access parameter; if index 2 i The corresponding first candidate off-chip memory access parameter decreases, then the value of i is updated, i is a value that gradually increases from 1; if index 2 i If the corresponding first candidate off-chip memory access parameter increases, the preset range is reduced to index [2 i-1 , 2 i-1], and searches for each first partition parameter in the narrowed range according to the search algorithm to execute steps 1032-1033.

[0095] In this example, the search space of the first partition parameter is narrowed down by an exponential search algorithm, so that the electronic device does not need to traverse all the first partition parameters within a preset range, thereby improving the efficiency of determining the first off-chip memory access parameter.

[0096] In other embodiments, the electronic device may also search for the first partition parameter based on other search algorithms to improve the search efficiency of the first partition parameter, such as searching based on a binary search method, etc. This embodiment does not limit the implementation method of the search algorithm.

[0097] Step 1032, determining the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter, and the first candidate off-chip memory access parameter.

[0098] After the input data scale of the operator is divided, the operation logic of the operator is not affected. Therefore, the electronic device determines the on-chip cache space corresponding to different data block scales in the calculation process based on the operation logic of the operator.

[0099] For example: Reference Figure 3 The operation logic of the MM operator shown is based on Figure 3 It can be seen that the input data of the operator are matrix A and matrix B. The input data scale shape of matrix A is [batch_size, M, K], the corresponding input data scale shape of matrix B is [K, N], and the input data scale shape of matrix C is [batch_size, M, N]. The operation logic of the MM operator includes: for the index value b of each batch from [0, batch_size-1], calculate the product of the value of the m+1th row and k+1th column in the matrix A of the b+1th batch and the value of the k+1th row and n+1th column in the matrix B, and determine the value of the m+1th row and n+1th column in the matrix C of the b+1th batch based on each product. Among them, m is the index value taken from [0, M-1]; n is the index value taken from [0, N-1]; k is the index value taken from [0, K-1].

[0100] Based on this operation logic, if all the data for initializing the MM operator is stored in an off-chip storage space, such as an off-chip DRAM, the data needs to be read from the off-chip to the on-chip buffer during the calculation process. Figure 4, if the first partition parameter is (sub_batch, sub_m, sub_n, sub_k), where sub_batch is used to partition batch_size, sub_m is used to partition M, sub_n is used to partition N, and sub_k is used to partition K. At this time, the data block size corresponding to matrix A is The data block size corresponding to matrix B is: Accordingly, matrix multiplication is performed based on the data block size corresponding to matrix A and the data block size corresponding to matrix B, and the data block size corresponding to matrix C is obtained as follows: in, Indicates that x is rounded up. Figure 3 From the operation logic shown, we can see that Figure 4 In , for each data block obtained by partitioning according to the first partitioning parameter, the calculation process of the data block corresponding to the matrix C is:

[0101] C[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1, n ite* sub_n:+sub_n-1]+=A[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1, k ite* sub_k:+sub_k-1]*B[k ite* sub_k: +sub_k-1, n ite* sub_n:+sub_n-1];

[0102] Among them, x:+y represents the interval [x, x+y], and x+=y represents x=x+y. That is, the above formula indicates that for the bth ite +1 batch block (m ite +1)×(k ite +1) matrix blocks, the batch index range [b ite *sub_batch, b ite *sub_batch+sub_batch-1], the batch and row index range corresponding to each batch index [m ite* sub_m,m ite* sub_m+sub_m-1], the value corresponding to each row index and the column index range [k ite* sub_k,k ite* sub_k+sub_k-1], the value corresponding to each column index is the same as the value corresponding to the (kthite +1)×(n ite +1) matrix blocks, the row index range [k ite* sub_k,k ite* sub_k+sub_k-1], the value corresponding to each row index and the column index range [m ite* sub_m,m ite* sub_m+sub_m-1], and determine the bth column in the C matrix based on each product. ite +1 batch block (m ite +1)×(n ite +1) values ​​of the matrix blocks.

[0103] according to Figure 4 It can be seen that b ite For The index value of the value, m ite For The index value of the value, n ite For The index value of the value, k ite For The index value to retrieve.

[0104] Afterwards, the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter is determined according to the calculation logic of the data block in combination with different cache mechanisms.

[0105] In one example, if the on-chip cache mechanism is a ping-pong cache, there are two buffers (usually referred to as buffer A and buffer B) in the ping-pong cache, and the two buffers are used to store input data and output data in turn. That is, when one buffer is used to store input data, the other buffer is used to store output data. According to the principle of ping-pong cache, both buffers of the on-chip cache need to store data blocks corresponding to input data, and one of the buffers needs to store data blocks corresponding to output data. Therefore, in Figure 4 In the data block scale shown, the on-chip cache space occupied by each data block during each calculation process is: 2*sub_batch*sub_m*sub_k+2*sub_k*sub_n+sub_batch*sub_m*sub_n.

[0106] In another example, if the on-chip cache mechanism is a single cache, a single cache means that there is only one buffer on the chip. In this case, the buffer needs to store data blocks corresponding to input data and data blocks corresponding to output data. Figure 4In the data block scale shown, the on-chip cache space occupied by each data block during each calculation process is: sub_batch*sub_m*sub_k+sub_k*sub_n+sub_batch*sub_m*sub_n.

[0107] In other embodiments, based on different on-chip cache mechanisms, the cache space occupied by each data block during each calculation process may also be other values, which will not be described in detail in this embodiment.

[0108] Optionally, in addition to the impact of the data block size on the off-chip memory access parameters, the loop order (i.e., data flow) of the data blocks read from the off-chip to the on-chip may also have an impact on the off-chip memory access parameters. In this case, different calculation methods also include calculations in different loop orders. The tensor information indicates that the input data size has i-dimensional information, and the calculation process of the indicating operator includes loop calculations for each dimensional information, where i is a positive integer. Accordingly,

[0109] Determine the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter, and the first candidate off-chip memory access parameter, including: determine the first candidate off-chip memory access parameter when calculating according to different loop orders between each dimensional information and the data block size corresponding to each first partition parameter.

[0110] In one example, the electronic device may determine, for each loop order, first candidate off-chip memory access parameters corresponding to different first partition parameters. At this time, determining the first candidate off-chip memory access parameters when calculating according to different loop orders between the dimensional information and the data block size corresponding to each first partition parameter includes:

[0111] For each loop sequence, from the i-th loop to the 1st loop, determine the data block size corresponding to different first partitioning parameters read from the off-chip to the on-chip in each calculation process of each loop layer, and the output data size obtained after each calculation; determine the off-chip memory access parameters corresponding to each loop layer based on the data block size and output data size corresponding to each calculation; determine the first candidate off-chip memory access parameters corresponding to each loop sequence based on the off-chip memory access parameters corresponding to each loop layer. At this time, the first candidate off-chip memory access parameters include the off-chip memory access parameters corresponding to each loop sequence.

[0112] Among them, one layer of loop refers to the process of traversing each value of the dimension information corresponding to the layer of loop and performing calculations based on each value; each calculation refers to the process of reading the data block size according to the current value of the dimension information corresponding to each layer of loop, and performing calculations based on the operation logic of the operator. Accordingly, since the dimension information corresponding to each layer of loop has multiple values, each layer of loop has multiple calculations.

[0113] That is, if there are i layers of loops, first iterate backwards from the i-th layer loop. The i-th layer loop needs to read in the data block when executing each on-chip calculation. At this time, the data block size of the input data and the data block size of the output data corresponding to different first partitioning parameters can be obtained. Then, determine the overall off-chip memory access parameters corresponding to the data block calculated by the i-th layer loop iteration, while the tensor data memory access parameters that are not affected remain unchanged. Then, jump to the i-1th layer loop, and continue to determine the overall off-chip memory access parameters corresponding to the data block calculated by the layer loop iteration until each layer iteration is completed.

[0114] for example: Figure 4 There are 4 layers of loops in the code, that is, i=4, and the first to fourth layers of loops are: b ite The cycle in which m ite The cycle in which n ite The loop in which k ite The loop where it is located. At this time, from the fourth layer k ite The cycle starts, k ite Each value of corresponds to a calculation, traversing k ite The calculation of the fourth loop is completed for all values ​​of . Each calculation needs to read in the input matrix with data block size of [sub_batch, sub_m, sub_k], [sub_k, sub_n], and output the output matrix with data block size of [sub_batch, sub_m, sub_n]. When iteration k ite In the loop, only two input matrices have k dimensions that need to be iterated. After the iteration, the data block size of the input matrix becomes [sub_batch, sub_m, K] and [K, sub_m, sub_n], and the data block size of the output matrix becomes [sub_batch, sub_m, sub_n]. ite In the loop, although the first input matrix does not have n dimensions, since the fourth layer K dimensions have been iterated, the first matrix also needs multiple iterations. Figure 4 The off-chip memory access parameter Memory access corresponding to the loop sequence shown (i.e., i=4, the first candidate off-chip memory access parameter of the loop sequence 4-1) is expressed by the following formula:

[0115]

[0116] If i=1, that is, the coefficients outside the brackets in the memory access calculation formula are 1, the first candidate off-chip memory access parameter corresponding to the case where there is no loop can be obtained.

[0117] In another example, the electronic device may also determine the first candidate off-chip memory access parameters corresponding to different loop orders for each first partition parameter. The calculation principle is the same as above. This embodiment does not limit the method for determining the first candidate off-chip memory access parameters.

[0118] Step 1033 , selecting the first off-chip memory access parameter from each first candidate off-chip memory access parameter whose on-chip cache space satisfies the on-chip cache constraint.

[0119] As can be seen from the above, satisfying the on-chip cache constraint means that the on-chip cache space occupied by the computing process is less than the on-chip cache constraint Buffer_size.

[0120] For example, the on-chip cache space corresponding to the above ping-pong cache is: sub_batch*sub_m*sub_k+sub_k*sub_n+sub_batch*sub_m*sub_n<Buffer_size, which satisfies the on-chip cache constraint.

[0121] For another example, the on-chip cache space corresponding to the above single cache is sub_batch*sub_m*sub_k+sub_k*sub_n+sub_batch*sub_m*sub_n<Buffer_size, which satisfies the on-chip cache constraint.

[0122] In one example, screening a first off-chip memory access parameter from among various first candidate off-chip memory access parameters whose on-chip cache space satisfies the on-chip cache constraint includes: determining the smallest first candidate off-chip memory access parameter from among various first candidate off-chip memory access parameters whose on-chip cache space satisfies the on-chip cache constraint, and obtaining the first off-chip memory access parameter.

[0123] In other embodiments, the method for screening the first off-chip memory access parameter may also be other methods. This embodiment does not limit the method for screening the first off-chip memory access parameter.

[0124] Step 104, for each group of operators in the multiple operators that have a fusion relationship, based on the tensor information of each operator in the operator group, determine the off-chip memory access parameters when the operator group is fused and calculated through different calculation methods, and obtain the second off-chip memory access parameters that meet the on-chip cache constraints.

[0125] In one example, a group of operators with a fusible relationship refers to: the same group of output results of a first operator is input into at least one second operator. In this case, the first operator and the second operator have a fusible relationship, and the second operator is also called the successor operator of the first operator. Figure 5 As shown, Figure 5Each circle in the diagram represents an operator, and the lines between operators represent activation values ​​that need to be transferred through off-chip storage. Figure 5 It can be seen that the output result 1 of operator A is input to operator B and operator C, and the output result 2 of operator A is input to operator D. At this time, operator A and operators B and C have a fusible relationship, that is, they can be used as a group of operator groups; operator A and operator D have a fusible relationship, that is, they can be used as a group of operator groups.

[0126] In another example, an operator group having a fusible relationship means that the same group of output results of a first operator is input to at least one second operator, and the type of the first operator and the type of the second operator match a preset fusible type combination.

[0127] For example, operators of a specific type have a high probability of fusion, so a fusionable type combination can be constructed based on the specific type, and the type combination consisting of the type of the first operator and the type of the second operator is matched with the fusionable type combination. If the match is successful, the first operator and the second operator form an operator group; if the match is unsuccessful, the first operator and the second operator do not form an operator group. At this time, for different operators with a low fusion probability, the off-chip memory access parameters after fusion are not required, which can save the computing resources of the electronic device.

[0128] A successful match means that the type of the first operator and the type of the second operator are the same as the types in the fusion type combination. Figure 6 The various fusible type combinations shown, the fusible type combinations and the first operator and the second operator that match each fusible type combination include:

[0129] A type combination of vector+element, where the first operator and the second operator matching the combination include: MV operator+activation function operator or residual module operator;

[0130] A combination of vector+vector types, where the first operator and the second operator matching the combination include: MV operator+bidirectional Softmax operator, or MV operator+bidirectional Norm operator, or MV operator+MV operator (where each MV operator can also be inserted with an element-wise operation);

[0131] A combination of matrix+element types, where the first operator and the second operator matching the combination include: MM operator or conv operator+activation function operator, or MM operator or conv operator+residual add operator;

[0132] A combination of matrix+vector types, where the first operator and the second operator matching the combination include: MM operator+bidirectional Softmax operator, or MM operator+bidirectional RSM Norm operator, or LayerNorm operator+conv operator;

[0133] A combination of matrix+matrix types, where the first operator and the second operator matching the combination include: MM operator+MM operator, or batch MM operator+Softmax operator (the Softmax operator may be a vector-level or element-level input conversion)+batch MM operator, or conv operator+MM operator, or conv operator+conv operator, or MM operator+conv operator, or MM operator+MM operator+LayerNorm operator;

[0134] A type combination of parallel matrices, wherein the first operator and the second operator matching the combination include: a query, key, value (QKV) vector, wherein the QKV vector may be an architecture based on an MM operator (i.e., MM base) or an architecture based on a conv operator (i.e., conv base), or a feed-forward neural network (FFN) operator, wherein the FFN operator may be an FFN layer in a large language model Llama;

[0135] A combination of matrix or vector+rearrangement type, where the first operator and the second operator matching the combination include: padding function+conv operator, or interpolate function+conv operator, or MM operator+transpose function, or MV operator+concat function, or MM operator+reshape function.

[0136] Figure 6 The fusion type combinations shown are only illustrative. In actual implementation, the fusion type combinations may include more or fewer types, which will not be listed one by one in this embodiment.

[0137] Optionally, the electronic device obtains each group of operator groups with a fusible relationship including: converting the data flow relationship between operators in the neural network into a graph network, wherein the operators serve as nodes in the graph network, and the data flow relationship between the operators serve as edges in the graph network; node features include but are not limited to: node type and tensor information of the node; edge features include but are not limited to: data flow relationship between nodes; and parsing the graph network to obtain each group of operator groups with a fusible relationship.

[0138] In other embodiments, the neural network may not be converted into a graph network, but the network information of the neural network may be directly parsed to obtain an operator group with a fusible relationship. This embodiment does not limit the method for obtaining the operator group with a fusible relationship.

[0139] In this embodiment, fusion calculation refers to a calculation method in which the output data of the first operator in the operator group is not written back to the off-chip storage, but is retained in the on-chip cache space for use by the second operator calculation. It can be seen that the on-chip cache space occupied during the operator fusion calculation process is larger than the on-chip cache space occupied by a single operator. Therefore, in order to ensure that the second off-chip memory access parameter in the fusion calculation process meets the on-chip cache constraint, it is necessary to divide the data scale of the operator group into blocks, thereby reducing the data scale of the operator group. At this time, the calculation method includes a block division method, and different block division methods correspond to different second division parameters.

[0140] Accordingly, for each group of operators in the plurality of operators that have a fusionable relationship, based on the tensor information of each operator in the operator group, off-chip memory access parameters when the operator group is fused and calculated by different calculation methods are determined to obtain second off-chip memory access parameters that meet the on-chip cache constraint, including steps 1041-1043:

[0141] Step 1041 , dividing the input data scale indicated by the tensor information of each operator in the operator group into data blocks according to different second division parameters, and obtaining the data block scale corresponding to each second division parameter.

[0142] The method for obtaining the second partition parameter refers to the above description of the first partition parameter, which will not be repeated here in this embodiment.

[0143] Step 1042, determining the on-chip cache space occupied when performing fusion calculation according to the data block size corresponding to each second partition parameter, and the second candidate off-chip memory access parameter.

[0144] According to the calculation logic of the data block and different cache mechanisms, the on-chip cache space occupied when calculating the data block size corresponding to each second partition parameter is determined.

[0145] Compared with the on-chip cache space corresponding to a single operator, the on-chip cache space occupied during fusion computing also needs to increase the data block size corresponding to the output data of the first operator.

[0146] For example, the first operator in the operator group is the MM operator, and the second operator is the element-wise add operator. The operation logic of the elementwise add operator is referenced Figure 7 As shown, according to Figure 7It can be seen that the input data of the operator are matrix A and matrix B. The input data scale shape of matrix A is [batch_size, M, N], the corresponding input data scale shape of matrix B is [batch_size, M, N], and the input data scale shape of matrix C is [batch_size, M, N]. The operation logic of the elementwise add operator includes: for each batch of index values ​​b from [0, batch_size-1], calculate the sum of the values ​​of the m+1th row and n+1th column in matrix A and the values ​​of the m+1th row and n+1th column in matrix B, and obtain the values ​​of the m+1th row and n+1th column in matrix C. Among them, m is the index value taken from [0, M-1] in sequence; n is the index value taken from [0, N-1] in sequence.

[0147] Combination Figure 4 The operation logic of the MM operator shown is Figure 7 The operation logic of the elementwise add operator shown in the figure can be used to obtain the operation logic of the MM operator + elementwise add operator fusion calculation. Figure 8 As shown, the first operator performs matrix multiplication operation based on input data blocks A and B to obtain D, which is used by the second operator. The second operator performs element-by-element addition operation based on D and input data block C to obtain updated D.

[0148] That is, for each data block divided according to the second division parameter, the calculation process of the data block corresponding to the matrix D is:

[0149] D[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1, n ite* sub_n:+sub_n-1]+=A[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1, k ite* sub_k:+sub_k-1]*B[k ite* sub_k: +sub_k-1, n ite* sub_n:+sub_n-1];

[0150] D[b ite *sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1, n ite* sub_n: +sub_n-1]+=C[b ite*sub_batch:+sub_batch-1,m ite* sub_m: +sub_m-1, n ite* sub_n:+sub_n-1].

[0151] Among them, x:+y represents the interval [x, x+y], and x+=y represents x=x+y. That is, the above formula indicates that for the bth ite +1 batch block (m ite +1)×(k ite +1) matrix blocks, the batch index range [b ite *sub_batch, b ite *sub_batch+sub_batch-1], the batch and row index range corresponding to each batch index [m ite* sub_m,m ite* sub_m+sub_m-1], the value corresponding to each row index and the column index range [k ite* sub_k,k ite* sub_k+sub_k-1], the value corresponding to each column index is the same as the value corresponding to the (kth ite +1)×(n ite +1) matrix blocks, the row index range [k ite* sub_k,k ite* sub_k+sub_k-1], the value corresponding to each row index and the column index range [m ite* sub_m,m ite* sub_m+sub_m-1], and determine the bth column in the D matrix based on each product. ite +1 batch block (m ite +1)×(n ite +1) matrix block value, the value of this matrix block is not written back off-chip.

[0152] Then, the bth ite +1 batch block (m ite +1)×(n ite +1) The value of the matrix block is the same as the value of the bth matrix in matrix C ite +1 batch block (m ite +1)×(n ite +1) matrix blocks are added element by element to obtain the updated D matrix. ite +1 batch block (m ite +1)×(n ite +1) values ​​of the matrix blocks.

[0153] according to Figure 8It can be seen that b ite For The index value of the value, m ite For The index value of the value, n ite For The index value of the value, k ite For The index value to retrieve.

[0154] Correspondingly, if the on-chip cache mechanism is a ping-pong cache, the on-chip cache space occupied by each data block during each calculation process is: 2*sub_batch*sub_m*sub_k+2*sub_k*sub_n+2sub_batch*sub_m*sub_n. Figure 4 Compared with the on-chip cache space corresponding to the single MM operator shown in FIG. 1 , the data block size of the output data block D of the first operator is increased by sub_batch*sub_m*sub_n.

[0155] If the on-chip cache mechanism is a single cache, the on-chip cache space occupied by each data block during each calculation is: sub_batch*sub_m*sub_k+sub_k*sub_n+2sub_batch*sub_m*sub_n. Figure 4 Compared with the on-chip cache space corresponding to the single MM operator shown in FIG. 1 , the data block size of the output data block D of the first operator is increased by sub_batch*sub_m*sub_n.

[0156] According to the above, during the fusion calculation process, the cyclic order (that is, data flow) of the data blocks read from the off-chip to the on-chip may also affect the off-chip memory access parameters. Based on this, the tensor information of each operator in the operator group indicates that the input data scale of the operator group has j-dimensional information, and the fusion calculation process of the operator group includes the cyclic calculation of each dimensional information, j is a positive integer; accordingly, the on-chip cache space occupied when the fusion calculation is performed according to the data block scale corresponding to each second partition parameter, and the second candidate off-chip memory access parameters are determined, including: determining the second candidate off-chip memory access parameters when the fusion calculation is performed according to the different cyclic orders between the dimensional information and the data block scale corresponding to each second partition parameter.

[0157] In one example, determining the second candidate off-chip memory access parameter when performing fusion calculation according to different loop orders between the dimensional information and the data block size corresponding to each second partition parameter includes:

[0158] For each loop sequence, from the j-th loop to the 1-th loop, determine layer by layer, in each fusion calculation process of each loop, the size of the first data block corresponding to the different second partitioning parameters read from the off-chip to the on-chip when executing the first operator in the operator group, and the size of the first output data obtained after the calculation;

[0159] Determine the first output data size corresponding to each second partition parameter, the second data block size corresponding to the same second partition parameter read from off-chip to on-chip when executing the second operator in the operator group, and the second output data size obtained after the calculation;

[0160] Determine the off-chip memory access parameter corresponding to each layer of loop based on the first data block size, the first output data size, the second data block size, and the second output data size corresponding to each calculation;

[0161] The second candidate off-chip memory access parameter corresponding to each loop sequence is determined based on the off-chip memory access parameter corresponding to each layer of loop.

[0162] for example: Figure 8 There are 4 layers of loops in the code, that is, i=4, and the 4th layer of loops from the 1st layer to the 4th layer are: b ite The cycle in which m ite The cycle in which n ite The loop in which k ite The loop where it is located. At this time, from the fourth layer k ite At the beginning of the loop, each calculation needs to read in the input matrix of the first data block size corresponding to the first operator from the off-chip, and the output matrix of the first output data size is [sub_batch, sub_m, sub_n]. After that, the second data block size corresponding to the second operator is read in from the off-chip, and the output matrix of the second output data size is [sub_batch, sub_m, sub_n]. Based on the first output data size and the second data block size, the output matrix of the second output data size is determined to be [sub_batch, sub_m, sub_n]. In summary, Figure 8 The off-chip memory access parameter Memoryaccess corresponding to the loop sequence shown is expressed by the following formula:

[0163]

[0164] Based on the above, it can be seen that compared with the single MM operator, the off-chip memory access parameter corresponding to the loop sequence in the fusion calculation process becomes larger.

[0165] Step 1043 , selecting the second off-chip memory access parameter from the second candidate off-chip memory access parameters whose on-chip cache space satisfies the on-chip cache constraint.

[0166] The method of screening the second off-chip memory access parameter is the same as the method of screening the first off-chip memory access parameter. For detailed description, please refer to the method of screening the first off-chip memory access parameter above, which will not be repeated in this embodiment.

[0167] Step 105 : For each operator group, determine the fusion benefit of the operator group based on the first off-chip memory access parameter of each operator in the operator group and the second off-chip memory access parameter of the operator group.

[0168] In one example, for each operator group, the fusion benefit of the operator group is determined based on the first off-chip memory access parameters of each operator in the operator group and the second off-chip memory access parameters of the operator group, including: determining the sum of the first off-chip memory access parameters of each operator in the operator group to obtain the parameter sum; determining the difference between the second off-chip memory access parameter and the parameter sum to obtain the fusion benefit of the operator group.

[0169] In other embodiments, the quotient between the second off-chip memory access parameter and the parameter sum may also be determined to obtain the fusion benefit of the operator group. This embodiment does not limit the method for determining the fusion benefit.

[0170] Step 106, when the fusion benefit of at least one group of operator groups meets the preset conditions, perform operator fusion on the operator group with the largest fusion benefit; take the fused operator group as one operator, and take the second off-chip memory access parameter corresponding to the fused operator group as the first off-chip memory access parameter, triggering the execution of each group of operator groups with a fusible relationship among multiple operators, based on the tensor information of each operator in the operator group, determining the off-chip memory access parameters when the operator group is calculated by different computing methods after fusion, and obtaining the second off-chip memory access parameter that meets the on-chip cache constraint and subsequent steps, that is, executing step 104.

[0171] In one example, if the fusion gain is the difference between the second off-chip memory access parameter and the sum of the parameters, accordingly, the fusion gain satisfies the preset condition that the fusion gain is greater than 0; in other embodiments, the fusion gain may be greater than a preset value, and the preset value is greater than 0. This embodiment does not limit the value of the preset value.

[0172] In another example, if the fusion gain is the quotient between the second off-chip memory access parameter and the parameter sum, accordingly, the fusion gain satisfies the preset condition that the fusion gain is greater than 1. In other embodiments, the fusion gain may be greater than a preset quotient value, and the preset quotient value is greater than 1. This embodiment does not limit the value of the preset quotient value.

[0173] In summary, the method for determining the operator fusion strategy of the neural network provided in this embodiment determines the off-chip memory access parameters of the operator when calculating in different calculation methods based on the tensor information of a single operator, and obtains the first off-chip memory access parameters that meet the on-chip cache constraint; for each group of operators with a fusionable relationship among multiple operators, based on the tensor information of each operator in the operator group, the off-chip memory access parameters when calculating in different calculation methods after the operator group is fused are determined, and the second off-chip memory access parameters that meet the on-chip cache constraint are obtained; for each group of operators, based on the first off-chip memory access parameters of each operator in the operator group The external memory parameters and the second chip external memory parameters of the operator group are used to determine the fusion benefit of the operator group; when the fusion benefit of at least one group of operator groups meets the preset conditions, the operator group with the largest fusion benefit is fused; the operator group with the largest fusion benefit is searched through the loop to determine the operator fusion strategy that meets the on-chip cache constraint; the operator fusion strategy can be determined without designing the hardware platform; the problem that the traditional operator fusion strategy cannot determine the fusion strategy without designing the hardware platform can be solved, thereby improving the versatility and flexibility of the operator fusion strategy determination method.

[0174] Optionally, based on the above embodiment, when the fusion benefits of each operator group do not meet the preset conditions, the design mode of the hardware platform is determined based on the first off-chip memory access parameters of each operator obtained after the operator fusion.

[0175] According to the above step 106, after the operator group is fused, the fused operator group is processed as one operator. Therefore, the operators obtained after the operator fusion include: the operators not involved in the fusion, and the operators corresponding to the fused operator group as a whole.

[0176] In one example, based on the first off-chip memory access parameter of each operator obtained after the operator fusion, a design method of the hardware platform is determined, including:

[0177] Based on the first off-chip memory access parameter of each operator obtained after operator fusion, the minimum off-chip memory access parameter corresponding to the on-chip cache constraint is determined; based on multiple on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each on-chip cache constraint, the optimal on-chip cache constraint of the hardware platform is determined.

[0178] Specifically, the sum of the first off-chip memory access parameters of each operator is calculated to obtain the minimum off-chip memory access parameter; the on-chip cache constraints are gradually increased, and the minimum off-chip memory access parameter corresponding to each on-chip cache constraint is determined according to the above embodiment. Generally, as the on-chip cache constraints increase, the corresponding minimum off-chip memory access parameter will decrease. However, after the on-chip cache constraints increase to a certain extent, the minimum off-chip memory access parameter will no longer continue to decrease, or the decrease will not be too large. After that, continuing to increase the on-chip cache constraints will not have a significant impact on the minimum off-chip memory access parameter. Based on this, based on multiple on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each on-chip cache constraint, the optimal on-chip cache constraint of the hardware platform is determined, including:

[0179] For the on-chip cache constraints sorted from small to large, and the minimum off-chip memory access parameters corresponding to each on-chip cache constraint, determine whether the difference between the minimum off-chip memory access parameters corresponding to two adjacent on-chip cache constraints is less than or equal to a preset difference range; if the difference is less than or equal to the preset difference range, determine that the smaller on-chip cache constraint of the two adjacent on-chip cache constraints is the optimal on-chip cache constraint.

[0180] or,

[0181] Based on various on-chip cache constraints and the minimum off-chip memory access parameters corresponding to each on-chip cache constraint, a relationship curve between the on-chip cache constraints and the minimum off-chip memory access parameters is generated; a curve point on the relationship curve where the absolute value of the slope is less than a preset absolute value is determined, and the on-chip cache constraint corresponding to the curve point is determined as the optimal on-chip cache constraint.

[0182] The optimal on-chip cache constraint may also be determined in other ways, which will not be described in detail in this embodiment.

[0183] In another example, based on the first off-chip memory access parameters of each operator obtained after the operator fusion, a design method of the hardware platform is determined, including:

[0184] A cost function is obtained, where the cost function is based on the relationship between the on-chip cache constraint and the hardware area of ​​the hardware platform; based on the first off-chip memory access parameter of each operator obtained after the operator fusion, the minimum off-chip memory access parameter corresponding to the on-chip cache constraint is determined; based on the cost function value corresponding to each cache constraint among the multiple on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each cache constraint, the optimal on-chip cache constraint is determined.

[0185] Since the hardware area of ​​the hardware platform is related to the manufacturing cost of the hardware, by obtaining the cost function and determining the optimal on-chip cache constraint based on the cost function value, the manufacturing cost of the hardware platform can be saved while ensuring smaller off-chip memory access parameters.

[0186] Optionally, the cost function may be sent by other devices, or may be obtained by fitting based on the corresponding relationship between each on-chip cache constraint and the hardware area. This embodiment does not limit the method for obtaining the cost function.

[0187] Based on a cost function value corresponding to each cache constraint among the multiple on-chip cache constraints and a minimum off-chip memory access parameter corresponding to each cache constraint, an optimal on-chip cache constraint is determined, including:

[0188] Input the cost function value corresponding to each on-chip cache constraint and the minimum off-chip memory access parameter into a preset mathematical model, and determine the on-chip cache constraint corresponding to the maximum value of the output result of the data model as the optimal on-chip cache constraint;

[0189] The data model is represented by the following formula:

[0190] 1 / (MemoryAccess*Cost)

[0191] Among them, MemoryAccess represents the minimum off-chip memory access parameter, and Cost represents the cost function value.

[0192] In other embodiments, the mathematical model may also be MemoryAccess*Cost. In this case, the on-chip cache constraint corresponding to the maximum value of the output result of the data model is determined as the optimal on-chip cache constraint. This embodiment does not limit the implementation method of the mathematical model and the determination method of the optimal on-chip cache constraint.

[0193] In this embodiment, the optimal on-chip cache constraint is determined by the relationship between the on-chip cache constraint and the minimum off-chip memory access parameter. The optimal on-chip cache constraint of the hardware platform can be determined before the hardware platform is designed, which not only avoids wasting cache space, but also ensures smaller off-chip memory access parameters and saves the production cost of the hardware platform.

[0194] Fig. 9 A block diagram of a device for determining an operator fusion strategy of a neural network according to an embodiment of the present disclosure is shown, the device comprising: an information acquisition module 910, a constraint acquisition module 920, a first determination module 930, a second determination module 940, a benefit determination module 950, and an operator fusion module 960.

[0195] An information acquisition module 910 is used to acquire tensor information of multiple operators in a neural network to be analyzed;

[0196] A constraint acquisition module 920, used to acquire on-chip cache constraints of the hardware platform to be designed;

[0197] A first determination module 930 is used to determine, for each operator, based on the tensor information of the operator, an off-chip memory access parameter when the operator is calculated according to different calculation methods, and obtain a first off-chip memory access parameter that satisfies the on-chip cache constraint;

[0198] A second determination module 940 is configured to determine, for each group of operators in the plurality of operators that have a fusionable relationship, off-chip memory access parameters when the operator group is fused and calculated using different calculation methods based on tensor information of each operator in the operator group, to obtain a second off-chip memory access parameter that satisfies the on-chip cache constraint;

[0199] A benefit determination module 950, configured to determine, for each operator group, a fusion benefit of the operator group based on a first off-chip memory access parameter of each operator in the operator group and a second off-chip memory access parameter of the operator group;

[0200] The operator fusion module 960 is used to fuse the operator group with the largest fusion benefit when the fusion benefit of at least one group of operator groups meets the preset conditions; take the fused operator group as an operator and the second off-chip memory access parameter corresponding to the fused operator group as the first off-chip memory access parameter, triggering the execution of the step of determining, for each group of operator groups with a fusible relationship among the multiple operators, the off-chip memory access parameter when the operator group is calculated by different calculation methods after the fusion of the operator group based on the tensor information of each operator in the operator group, and obtaining the second off-chip memory access parameter that meets the on-chip cache constraint and subsequent steps.

[0201] For related description, please refer to the above method embodiment.

[0202] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0203] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0204] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0205] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0206] Fig.10 1 is a block diagram of a device 1900 for determining an operator fusion strategy of a neural network according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Fig.10 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0207] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0208] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0209] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining an operator fusion strategy of a neural network, characterized in that: The method comprises: Obtain tensor information of multiple operators in the neural network to be analyzed; Obtain on-chip cache constraints of the hardware platform to be designed; For each operator, based on the tensor information of the operator, determine the off-chip memory access parameter of the operator when performing calculations in different calculation modes, and obtain a first off-chip memory access parameter that satisfies the on-chip cache constraint; For each group of operators in the plurality of operators that have a fusionable relationship, based on tensor information of each operator in the operator group, determining off-chip memory access parameters when the operator group is fused and calculated by different calculation methods, to obtain a second off-chip memory access parameter that satisfies the on-chip cache constraint; For each operator group, determining a fusion benefit of the operator group based on a first off-chip memory access parameter of each operator in the operator group and a second off-chip memory access parameter of the operator group; When the fusion benefit of at least one group of operator groups meets the preset conditions, the operator group with the largest fusion benefit is subjected to operator fusion; the fused operator group is taken as one operator, and the second off-chip memory access parameter corresponding to the fused operator group is taken as the first off-chip memory access parameter, triggering the execution of the step of determining, for each group of operator groups having a fusible relationship among the multiple operators, the off-chip memory access parameter when the operator group is calculated by different computing methods after fusion based on the tensor information of each operator in the operator group, and obtaining the second off-chip memory access parameter that meets the on-chip cache constraint and subsequent steps.

2. The method according to claim 1, characterized in that The step of determining, for each operator, an off-chip memory access parameter when the operator is calculated in different calculation modes based on the tensor information of the operator, and obtaining a first off-chip memory access parameter that satisfies the on-chip cache constraint includes: According to different first division parameters, the input data size indicated by the tensor information is divided into data blocks to obtain a data block size corresponding to each first division parameter; Determine the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter, and the first candidate off-chip memory access parameter; The first off-chip memory access parameter is selected from each first candidate off-chip memory access parameter whose on-chip cache space satisfies the on-chip cache constraint.

3. The method according to claim 2, characterized in that The tensor information indicates that the input data scale has i-dimensional information, and indicates that the calculation process of the operator includes a cyclic calculation of each dimensional information, where i is a positive integer; Accordingly, The determining of the on-chip cache space occupied when calculating the data block size corresponding to each first partition parameter and the first candidate off-chip memory access parameter includes: Determine the first candidate off-chip memory access parameter when calculating according to different loop orders between the dimensional information and the data block size corresponding to each first partition parameter.

4. The method according to claim 3, characterized in that The determining of the first candidate off-chip memory access parameter when calculating according to different loop orders between the dimensional information and the data block size corresponding to each first partition parameter includes: For each loop sequence, from the i-th loop to the 1-th loop, determine layer by layer the data block size corresponding to the different first partitioning parameters read from the outside to the chip during each calculation process of each loop, and the output data size obtained after each calculation is completed; Based on the data block size and output data size corresponding to each calculation, determine the off-chip memory access parameters corresponding to each layer of loop; A first candidate off-chip memory access parameter corresponding to each loop sequence is determined based on the off-chip memory access parameters corresponding to each layer of loop.

5. The method according to claim 1, characterized in that The step of determining, for each group of operators in the plurality of operators that have a fusionable relationship, off-chip memory access parameters when the operator group is fused and calculated by different calculation methods based on tensor information of each operator in the operator group, and obtaining a second off-chip memory access parameter that satisfies the on-chip cache constraint includes: According to different second partitioning parameters, the input data scale indicated by the tensor information of each operator in the operator group is partitioned into data blocks to obtain a data block scale corresponding to each second partitioning parameter; Determine the on-chip cache space occupied when performing fusion calculation according to the data block size corresponding to each second partition parameter, and the second candidate off-chip memory access parameter; The second off-chip memory access parameter is screened from each second candidate off-chip memory access parameter whose on-chip cache space satisfies the on-chip cache constraint.

6. The method according to claim 5, characterized in that The tensor information of each operator in the operator group indicates that the input data scale of the operator group has j-dimensional information, and indicates that the fusion calculation process of the operator group includes a cyclic calculation of each dimensional information, and j is a positive integer; accordingly, The determining of the on-chip cache space occupied when performing fusion calculation according to the data block size corresponding to each second partition parameter and the second candidate off-chip memory access parameter includes: Determine the second candidate off-chip memory access parameter when performing fusion calculation according to different loop orders between the dimensional information and the data block size corresponding to each second partition parameter.

7. The method according to claim 6, characterized in that The determining of the second candidate off-chip memory access parameter when performing fusion calculation according to different loop orders between the dimensional information and the data block size corresponding to each second partition parameter includes: For each loop sequence, from the j-th loop to the 1-th loop, determine layer by layer, in each fusion calculation process of each loop, the first data block size corresponding to the different second partitioning parameters read from the outside to the chip when executing the first operator in the operator group, and the first output data size obtained after the calculation; Determine the first output data scale corresponding to each second partition parameter, the second data block scale corresponding to the same second partition parameter read from off-chip to on-chip when executing the second operator in the operator group, and the second output data scale obtained after calculation; Determine the off-chip memory access parameter corresponding to each layer of loop based on the first data block size, the first output data size, the second data block size, and the second output data size corresponding to each calculation; The second candidate off-chip memory access parameter corresponding to each loop sequence is determined based on the off-chip memory access parameter corresponding to each layer of loop.

8. The method according to any one of claims 1 to 7, characterized in that: The step of determining, for each operator group, a fusion benefit of the operator group based on a first off-chip memory access parameter of each operator in the operator group and a second off-chip memory access parameter of the operator group includes: Determine the sum of the first off-chip memory access parameters of each operator in the operator group to obtain a parameter sum; The difference between the second off-chip memory access parameter and the sum of the parameters is determined to obtain a fusion benefit of the operator group.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: When the fusion benefits of each operator group do not meet the preset conditions, the design mode of the hardware platform is determined based on the first off-chip memory access parameters of each operator obtained after the operator fusion.

10. The method according to claim 9, characterized in that The determining of the design mode of the hardware platform based on the first off-chip memory access parameter of each operator obtained after the operator fusion includes: Determine the minimum off-chip memory access parameter corresponding to the on-chip cache constraint based on the first off-chip memory access parameter of each operator obtained after the operator fusion; Based on multiple on-chip cache constraints and minimum off-chip memory access parameters corresponding to each on-chip cache constraint, an optimal on-chip cache constraint of the hardware platform is determined.

11. The method according to claim 9, characterized in that The determining of the design mode of the hardware platform based on the first off-chip memory access parameter of each operator obtained after the operator fusion includes: Obtaining a cost function, the cost function being based on a relationship between the on-chip cache constraint and a hardware area of ​​the hardware platform; Determine the minimum off-chip memory access parameter corresponding to the on-chip cache constraint based on the first off-chip memory access parameter of each operator obtained after the operator fusion; An optimal on-chip cache constraint is determined based on a cost function value corresponding to each cache constraint among the multiple on-chip cache constraints and the minimum off-chip memory access parameter corresponding to each cache constraint.

12. A device for determining an operator fusion strategy of a neural network, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 11 when executing the instructions stored in the memory.

13. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images