Compilation method for neural network model and related product

Abstract

The present disclosure discloses a compilation method for a neural network model and related products, including a compiler, a device and a computer readable storage medium. The device can be included in a computing processing apparatus of a combined processing apparatus, which can include one or more data processing apparatuses. The aforementioned combined processing apparatus can also include an interface apparatus and other processing apparatuses. The computing processing apparatus interacts with the other processing apparatuses to jointly complete a user-specified computing operation. The combined processing apparatus can also include a storage apparatus connected to the device and the other processing apparatuses, respectively, for storing data of the device and the other processing apparatuses. By utilizing the scheme of the present disclosure, a reasonable distribution of scalar operations and tensor operations can be effectively achieved.

Description

Technical Field

[0001] This disclosure generally relates to the field of compilation. More specifically, this disclosure relates to a compilation method for neural network models, a compiler, an apparatus for compiling neural network models, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, neural network algorithms, as one of the research hotspots in this field, have been widely applied. To efficiently execute neural network algorithms, artificial intelligence processors (AI processors) are typically used for acceleration. During computation, neural network algorithms generate a large number of scalar and tensor operations. However, current neural network hardware architectures are not optimized for these scalar and tensor operations, resulting in low efficiency in processing them. Specifically, there is currently no effective solution to reasonably distribute the various operations of the neural network model across hardware devices to optimize the model's inference performance. Furthermore, due to the lack of corresponding optimization, neural network computations are not adequately accelerated, thus hindering the overall performance improvement of the computing system. Summary of the Invention

[0003] In view of the technical problems mentioned in the background section above, this disclosure provides a scheme for optimizing compilation. Using the scheme of this disclosure, the operations involved in the neural network model can be divided and evaluated during the compilation process, and the operations can be allocated to different types of processors for execution based on the evaluation results. Therefore, the scheme of this disclosure can accelerate the operation of neural networks and improve the overall performance of the neural network computing system. Based on this, this disclosure provides a scheme for optimizing compilation in the following aspects.

[0004] In a first aspect, this disclosure provides a method for compiling a neural network model, comprising: performing computational analysis on the neural network model to obtain tensor operations and scalar operations to be performed when executing the neural network model; evaluating the performance metrics of the tensor operations and scalar operations when executed on multiple processors; and allocating the tensor operations and scalar operations to corresponding processors for execution based on the evaluation results.

[0005] In a second aspect, this disclosure provides a compiler, comprising: a computation analysis unit configured to perform computation analysis on a neural network model to obtain tensor and scalar operations to be performed when executing the neural network model; and a computation allocation unit configured to: evaluate performance metrics of the tensor and scalar operations when executed on multiple processors; and allocate the tensor and scalar operations to corresponding processors for execution based on the evaluation results of the evaluation.

[0006] In a third aspect, this disclosure provides an apparatus for compiling a neural network model, comprising: a processor; and a memory storing a program that, when executed by the processor, causes the apparatus to perform the aforementioned compilation method and several embodiments thereof, which will be described below.

[0007] In a fourth aspect, this disclosure provides a computer-readable storage medium having a program stored thereon for compiling a neural network model, which, when executed by one or more processors, enables the aforementioned compilation method and several embodiments thereof, which will be described below, to be implemented.

[0008] Using the compilation scheme provided in this disclosure, the operations of a neural network model can be divided into tensor operations and scalar operations, so as to rationally allocate them to suitable processors (e.g., general-purpose processors or intelligent processors) for execution. During the allocation process, this disclosure evaluates the performance metrics of tensor and scalar operations running on the processor, and then allocates them based on the evaluation results. Based on this, the compilation optimization scheme of this disclosure can be used to accelerate neural network operations to achieve a good speedup ratio and also improve the overall performance of the computing system. Furthermore, due to the introduction of performance metrics, the allocation of tensor and scalar operations to appropriate processors is more accurate, thereby significantly reducing the cost and power consumption ratio of neural network operations. When applied to heterogeneous computing systems including hosts (which include general-purpose processors) and devices (which include intelligent processors), the scheme disclosed in this disclosure can rationally allocate tensor and scalar operations within the computing system, thereby optimizing the operational performance of the computing system and improving its computational efficiency. Attached Figure Description

[0009] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0010] Figure 1 This is a flowchart illustrating a method for compiling a neural network model according to an embodiment of the present disclosure;

[0011] Figure 2 This is a simplified block diagram illustrating a compiler according to an embodiment of the present disclosure;

[0012] Figure 3 This is a detailed block diagram illustrating a compiler according to an embodiment of the present disclosure;

[0013] Figure 4 This is a schematic block diagram illustrating an arithmetic allocation unit according to an embodiment of the present disclosure;

[0014] Figure 5 This is a structural diagram illustrating a combined processing apparatus according to an embodiment of the present disclosure; and

[0015] Figure 6 This is a schematic diagram illustrating the structure of a circuit board according to an embodiment of this disclosure. Detailed Implementation

[0016] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0017] Given the inefficiency of scalar and tensor operations in current neural network models, this disclosure proposes separating scalar and tensor operations during the neural network model compilation process, evaluating the performance metrics of scalar and tensor operations on the processor, and allocating scalar and tensor operations to appropriate processors based on the evaluation results. In one embodiment, the evaluation of the aforementioned performance metrics can be based on a cost function and / or a reward function. In another embodiment, the aforementioned processor can include various types of processors, including general-purpose processors (CPUs) or graphics processing units (GPUs). In one implementation scenario, when the aforementioned processor is a multi-core processor, the solution of this disclosure can allocate scalar or tensor operations to appropriate processor cores.

[0018] Figure 1 This is a flowchart illustrating a compilation method 100 for a neural network model according to an embodiment of the present disclosure. Figure 1 As shown, in step S102, the neural network model is analyzed to obtain the tensor operations (“Tensor Op”) and scalar operations (“Scalar Op”) to be performed when executing the neural network model. In one embodiment, the neural network model can be analyzed to obtain the aforementioned two operations based on the intermediate representation of the neural network model obtained through compilation.

[0019] Regarding the neural network model disclosed herein, structurally, it typically includes an input layer, an output layer, and hidden layers located between the input and output layers. Given training samples, the number of nodes in the input and output layers of the neural network is determined, while the number of hidden layers and neurons can be adjusted. In terms of representation, the neural network model can also be represented as a computational graph. In one implementation scenario, this computational graph can include multiple operators (OPs) and "edges" connecting the operators, where edges represent the dependencies between operators during execution or the data transfer relationships between operators. The operators are obtained by encapsulating a large number of common operations in the neural network, such as convolution operators, max-pooling operators, or ReLU operators (i.e., activation function operators). In the context of this disclosure, the aforementioned operators can include operators for performing tensor operations and operators for performing scalar operations.

[0020] Regarding the aforementioned intermediate representation, as those skilled in the art know, compilers are typically divided into front-end and back-end. During compilation, the front-end performs various analyses on the input program, including lexical analysis, syntax analysis, and semantic analysis, and then generates an intermediate representation (IR) resembling a data structure. Subsequently, the compiler's back-end optimizes the IR and generates the target code. The compiler in this embodiment can refer to a neural network compiler, such as TVM. This neural network compiler can receive a neural network model from a neural network programming framework (such as Tensorflow, PyTorch, or Caffe). Next, the compiler front-end first parses and reconstructs the neural network to obtain the computation graph mentioned above. Then, the compiler's optimizer can be used to perform optimization operations on the generated computation graph, such as fusion (e.g., fusion of multiple operators) or pruning (e.g., removing edges and operators in the computation graph that are irrelevant to the final output node), to obtain a graph-form intermediate representation, i.e., the aforementioned "IR". This graph-form intermediate representation can then be converted into a hardware-compatible intermediate representation. Finally, the code that can be executed on the hardware can be generated based on the intermediate representation adapted to the hardware, that is, the operation performed by the code generation unit disclosed herein.

[0021] Furthermore, the tensor operations obtained above may include, but are not limited to, vector operations and matrix operations (such as convolution); scalar operations may involve parameter operations, control operations, and loop count scheduling. In one embodiment, performing computational analysis to obtain the tensor and scalar operations to be performed may include compiling the neural network model according to the hardware configuration of the neural network model to obtain the tensor and scalar operations to be performed when executing the neural network model. Here, the hardware configuration of the neural network model may involve the number and distribution of processors in the computing system that computes the neural network model, the supported computational granularity (e.g., computation of 8-bit, 16-bit, 32-bit, 64-bit, or higher bit data), and the on-chip and off-chip data storage arrangements.

[0022] In one embodiment, the compilation described above may involve performing one or more of tensor shape derivation, tensor partitioning, and tensor optimization on the neural network model to obtain the tensor operations to be performed when executing the neural network model. Further, the tensor partitioning process inevitably generates some scalar operations, such as loop operations. Additionally, in one embodiment, the compilation may also include scalar optimization of scalar operations to obtain the scalar operations to be performed when executing the neural network model. The scalar operations to be performed when executing the neural network model may include, but are not limited to, scalar operations contained in the neural network model, and scalar operations generated during tensor partitioning. In one scenario, this disclosure also proposes scalar optimization of the scalar operations generated during the compilation process, performed on one or more of the shape derivation, partitioning, and optimization. The shape derivation, partitioning, and optimization operations will be discussed later in conjunction with... Figure 3 Provide a detailed description.

[0023] Next, in step S104, the performance metrics of the tensor and scalar operations performed on multiple processors are evaluated. In the context of this disclosure, processors can be general-purpose processors and intelligent processors (e.g., "GPUs") included in heterogeneous architecture systems. In some scenarios, when the intelligent processor is a multi-core processor, the solution of this disclosure also considers evaluating the performance metrics of the tensor and scalar operations performed on each core of the multi-core processor. Regarding the aforementioned performance metrics, they may, for example, relate to the gains or losses obtained in terms of computation time, complexity, efficiency, and data transfer when tensor or scalar operations are performed on a particular processor. Based on this, this disclosure proposes using a cost function and a benefit function to evaluate the performance metrics of tensor and scalar operations, wherein the aforementioned cost function can be used to determine the cost incurred by tensor or scalar operations running on a candidate processor, and the benefit function can be used to determine the gains incurred by tensor or scalar operations running on a candidate processor.

[0024] In one embodiment, the aforementioned cost function may include data transfer cost and / or cumulative cost, wherein the data transfer cost may include the cost incurred by transferring data when the tensor operation or scalar operation is run on the candidate processor, and the cumulative cost can be obtained by accumulating the cost incurred by the tensor operation or scalar operation continuously running on the candidate processor. Further, in one embodiment, the benefit function may include acceleration benefit and / or complexity benefit, wherein the acceleration benefit may be the computational efficiency benefit obtained by the tensor operation or scalar operation running on the candidate processor, and the complexity benefit is the benefit obtained when the computational complexity of the tensor operation or scalar operation is reduced when running on the candidate processor.

[0025] In one implementation scenario, the above cost function can be expressed as follows:

[0026] C(i,j)=T(i,j)+A(i,j) (1)

[0027] Where C(i,j) is the cost function mentioned above, T(i,j) represents the cost of data transfer when the i-th operation ("op[i]") runs on the j-th processor or processor core ("core[j]"), and A(i,j) represents the cumulative cost when the i-th operation runs on the j-th processor or processor core.

[0028] Depending on the scenario, the cost of the aforementioned data transmission can involve the type and size of the data being processed, and whether it involves control flow. Taking a scenario with control flow as an example, in a heterogeneous architecture including general-purpose processors and intelligent processors, statements outside the control flow (which involve one or more operators for scalar operations) can be computed by the general-purpose processor before the kernel function is launched, and can be passed to the intelligent processor, for example, via kernel parameters, so that the intelligent processor can use them when performing operations on other operators (such as tensor operators). However, for control flow (e.g., if and for statements), when the kernel function is run by the intelligent processor, the expressions in the control flow statements are only determined when the intelligent processor executes the basic block of that statement. For scalar operations in the control flow statements, the kernel function's computation on the intelligent processor needs to be stopped, and the data needs to be transferred to the general-purpose processor to perform the aforementioned scalar operations. The result of the scalar operation is then passed back to the intelligent processor so that the intelligent processor can perform subsequent operations based on the result. Therefore, control flow incurs a significantly higher transmission cost than regular data.

[0029] In one implementation scenario, the above-mentioned revenue function can be expressed as follows:

[0030] P(i,j)=S(i,j)+O(i,j) (2)

[0031] Here, P(i,j) is the benefit function mentioned above, S(i,j) represents the computational efficiency gain obtained by the i-th operation ("op[i]") running on the j-th processor or processor core ("core[j]"), and O(i,j) represents the gain obtained when the computational complexity is reduced when the i-th operation ("op[i]") runs on the j-th processor or processor core ("core[j]"). In terms of computational efficiency gains, general-purpose processors have an advantage over intelligent processors when processing scalar data, thus achieving significant benefits. For example, for the aforementioned scalar computations outside the control flow, the general-purpose processor needs to pass parameters to the intelligent processor after computation so that the intelligent processor can use the results of these scalar computations to perform tensor operations on the relevant tensor operators. It can be seen that, for scalar operations outside the control flow, operations on general-purpose processors can achieve reduced transmission costs and increased computational efficiency gains. In one implementation scenario, O(i,j) here can refer to the gain obtained by scalar or tensor operations running on an intelligent processor or intelligent processor core. The benefits of this reduced computational complexity can involve the complexity of the operators (e.g., multiplication is more complex than addition) as well as the amount and type of computation (e.g., multiplication of floating-point data is more complex than multiplication of integer data).

[0032] After evaluating tensor and scalar operations in the above manner, in step S106, tensor and scalar operations are assigned to corresponding processors for execution based on the evaluation results. In one embodiment, cost functions and benefit functions can be used to find function libraries (such as...). Figure 4 As shown in the diagram (described in detail later), the cost and benefit values ​​of the cost and benefit functions are obtained. Then, based on the cost and benefit values, it can be determined whether to assign tensor or scalar operations to one of the appropriate general-purpose or intelligent processors.

[0033] In one implementation, a first threshold can be set for the profit function, and a second threshold can be set for the cost function. Tensor operations or scalar operations can be assigned to one of the corresponding general-purpose processors or intelligent processors based on a comparison between the profit value and the first threshold and / or a comparison between the cost value and the second threshold. Specifically, when the profit value is greater than or equal to the first threshold, the corresponding tensor operation or scalar operation can be assigned to one of the corresponding general-purpose processors or intelligent processors. Additionally, when the cost value is less than or equal to the second threshold, the corresponding tensor operation or scalar operation can be assigned to one of the corresponding general-purpose processors or intelligent processors. Alternatively, when the profit value is greater than or equal to the first threshold and the cost value is less than or equal to the second threshold, the corresponding tensor operation or scalar operation can be assigned to one of the corresponding general-purpose processors or intelligent processors.

[0034] In one implementation scenario, the aforementioned intelligent processor may include multiple intelligent processor cores, such as quad-core, octa-core, sixteen-core, or more. As an example, each intelligent processor core may include multiple computational units to perform tensor or scalar operations according to the computational requirements of the neural network model. For these multiple intelligent processor cores, the disclosed solution can allocate tensor or scalar operations to one of the corresponding intelligent processor cores based on the aforementioned benefit value and / or cost value. In other words, the disclosed solution also supports computational task allocation at the intelligent processor core level.

[0035] The above combination Figure 1 The compilation method disclosed herein is described in detail. Using this method, computations in a neural network model can be specifically allocated to computing hardware (such as the general-purpose processors and intelligent processors mentioned above), thereby significantly optimizing the computational performance of the hardware platform running the neural network model. Furthermore, through the aforementioned computational allocation, the hardware platform can also achieve high performance in terms of power efficiency and speedup. Additionally, for variable neural network models, the solution disclosed herein can optimize the large-scale scalar operations that may occur in the network, thus further optimizing and improving the performance of the hardware platform.

[0036] Figure 2 This is a simplified block diagram illustrating a compiler 200 according to an embodiment of the present disclosure. It should be understood that the compiler 200 can perform the above-described combinations. Figure 1 The described compilation method, therefore regarding Figure 1 The description also applies to the specific details about the compiler described below. Furthermore, in heterogeneous architectures including general-purpose processors and intelligent processors, the compiler 200 can be deployed on a general-purpose CPU to execute, for example, the combinations described above. Figure 1 The compilation methods discussed.

[0037] As an optional embodiment, the compiler 200 may be a neural network compiler for compiling a neural network received from a neural network framework into target code capable of running on at least one processor. The compiler may include a compiler front-end for converting the neural network model into a graph-based intermediate representation and lower-level intermediate representations below the graph-based intermediate representation. Figure 2 As shown, the compiler 200 may further include a computation analysis unit 201 and a computation allocation unit 202. The computation analysis unit 201 and the computation allocation unit 202 are used to analyze the computations in the neural network model according to the intermediate representation, so that the computations in the neural network model can be allocated to at least one processor for execution, and to generate target code corresponding to the corresponding processor.

[0038] In one embodiment, the computation analysis unit can be configured to perform computation analysis on the neural network model to obtain the tensor and scalar operations to be performed when executing the neural network model. It can be seen that the computation analysis unit here performs... Figure 1 Step S102 in method 100. Accordingly, in one embodiment, the operation allocation unit may be configured to evaluate the performance metrics of the aforementioned tensor operations and scalar operations when executed on multiple processors, and to allocate the tensor operations and scalar operations to the corresponding processors for execution based on the evaluation results. Similarly, the operation allocation unit here performs the following: Figure 1 Steps S104 and S106 in method 100.

[0039] Although not shown in the figure, in one embodiment, compiler 200 may further include a code generation unit connected to the operation allocation unit. In operation, the code generation unit can generate corresponding code for scalar and tensor operations allocated by the operation allocation unit, so that the allocated operations can be executed accordingly by the general-purpose processor and the intelligent processor. For data-parallel operation scenarios, the code generation unit may include an identifier (“ID”) of the processor or processor core in the generated code.

[0040] As can be seen from the above description, the compiler 200 can achieve the following through its included operation analysis unit 201 and operation allocation unit 202: Figure 1 The steps included in Method 100 optimize the hardware platform for running the neural network model by separating tensor operations and scalar operations, thereby improving the computational efficiency of the neural network model and enhancing the overall performance of the hardware platform.

[0041] Figure 3 This is a detailed block diagram illustrating a compiler according to embodiments of the present disclosure. It will be understood that... Figure 3 The compiler shown is Figure 2The compiler shown is a specific implementation, and the same components are represented by the same numbers, therefore regarding Figure 2 The description of the compiler also applies. Figure 3 The compiler shown.

[0042] like Figure 3 As shown, the compiler includes an operation analysis unit 201 and an operation allocation unit 202, that is... Figure 2 The same unit is shown in the figure. As an example, the operation analysis unit includes an operation traversal unit 203. In one implementation scenario, as described above, the operation traversal unit can traverse the intermediate representations corresponding to the neural network model during the compilation process, thereby determining the tensor operations and scalar operations in the neural network model, namely the "Tensor Op" and "Scalar Op" shown in the figure, respectively.

[0043] For tensor operations, the operation analysis unit also includes a shape derivation unit 204, a block division unit 205, and an optimization unit 206. These three units can constitute the compilation unit in the context of this disclosure. In one implementation scenario, for a variable neural network model (e.g., whose input tensor data shape is variable), since the model runs layer by layer during execution, the shape of each layer's input can only be obtained at runtime. Therefore, during compilation, the shape derivation unit of this disclosure can express the tensor shape of each layer's input according to a certain expression.

[0044] For example, the computation of the first layer of a variable neural network is relatively simple; the shape of the input tensor can be determined simply by receiving the shape parameters of the input. The input to the second layer is obtained from the input and operations of the first layer (e.g., convolution operations in convolutional layers). After the operations of the first layer, the shape of the tensor data changes; that is, this shape change is determined by the Tensor Ops (e.g., convolution operations) of the first layer. Based on this principle, the shape derivation unit can express the input tensor of the second layer of the neural network with a certain expression, thus completing the shape derivation for the input of the second layer. This process continues until the shape derivation unit expresses the tensor shapes of the input and output tensors of each layer with a certain expression, thereby completing the shape derivation operation. Therefore, it can be understood that the output shape expression of the current layer in the neural network model depends on the shape expression of its input tensor and the tensor operations performed by the current layer. Taking convolution operations as an example, the output expression of this shape derivation could be, for example, the following formula:

[0045]

[0046] Regarding the aforementioned block unit 205, since the storage space of the hardware architecture (e.g., system-on-a-chip) executing the neural network model is limited, for large tensor operations (such as convolution operations between large tensors), the tensor data needs to be block-based before execution on the system-on-a-chip, i.e., the tensor data needs to be split. Taking the large matrix multiplication operation of a two-dimensional tensor as an example, if the system-on-a-chip only supports multiplication of matrix blocks of size 8*8, then for a 16*16 matrix, it needs to be block-based to adapt to the on-chip cache and computation requirements. It can also be understood that the splitting operation performed by the block unit takes into account the hardware configuration for running the neural network model, such as the storage space of the system-on-a-chip (e.g., cache size) and / or the computing power of the processor core. Since the block unit needs to calculate the expression generated by the shape derivation unit during the block-splitting process, and thus needs to execute loop operations (e.g., "for" loop statements), a large number of expressions will be generated during this period. These expressions can include scalar expressions, such as the specific number of blocks. Taking the convolution operation of the convolutional layer as an example, in order to make it suitable for performing convolution operations on the on-chip system, the expression (generated by the shape derivation unit) that serves as the input to the convolutional layer will be divided into blocks by the block unit to generate another expression.

[0047] The aforementioned optimization unit 206 is used to optimize the computation graph or IR. Depending on the implementation scenario, this optimization may include, for example, storage reuse of on-chip system data frequently handled by the compiler, pipelined operations (e.g., simultaneous execution of instructions involving copying subsequent data and calculating previous data), and fusion (including residing tensor data on-chip instead of loading it off-chip). During the aforementioned optimization operations performed by the optimization unit, a large number of scalar operations are also generated. For example, during fusion, when combining the "for" statements of the previous and subsequent block operations, it involves determining how many data blocks are there and where these data blocks are moved, which inevitably involves scalar computation, thus generating a certain amount of scalar operations.

[0048] After the operations of the shape derivation unit 204, the block division unit 205 and the optimization unit 206, the optimized tensor operation will be passed to the operation allocation unit 202, so that the tensor operation can be allocated to the general-purpose processor 208 or the intelligent processor core 209 shown in the figure for execution.

[0049] The optimization unit 207 for scalar operations, further illustrated in the figure, will be described below. In one implementation scenario, the optimization unit 207 can perform scalar simplification, constant passing, and / or dead code elimination for scalar operations to simplify intermediate expressions and thereby reduce the amount of code. Taking constant passing as an example, for the previous expression A = 4 and the next expression C = A + B, the optimization unit can optimize the next expression to C = 4 + B. Taking dead code elimination as another example, for a tensor shape generated by the shape derivation unit, when it is no longer used in subsequent calculations, the code associated with that shape can also be directly deleted. As shown by arrow 210, the optimization unit 207 can also receive various scalar operations generated by the shape derivation unit, the block unit, and the optimization unit 206 during operation, and also perform similar optimization operations as described above to pass the optimized scalar operations to the operation allocation unit 202. Subsequently, the optimized scalar operation will be passed to the operation allocation unit 202, which will then allocate the scalar operation to the general-purpose processor 208 or the intelligent processor core 209 shown in the figure for execution.

[0050] In one implementation scenario, the computation allocation unit can assign high-complexity scalar operations to general-purpose processors, while assigning complex tensor operations and general-complexity scalar operations to intelligent processors. In another implementation scenario, the computation allocation unit can also uniformly assign scalar operations to general-purpose processors and tensor operations to intelligent processors, i.e., uniformly allocate based on the type of operation. The following will combine... Figure 4 The operating principle of the arithmetic allocation unit 202 is described in detail.

[0051] Figure 4 This is a schematic block diagram illustrating a computation allocation unit 202 according to an embodiment of the present disclosure. As previously described, the computation allocation unit disclosed herein can be configured to evaluate tensor operations and scalar operations using the value of a cost function (the C(i,j) value shown in the figure) and / or a benefit function (the P(i,j) value shown in the figure). The cost function is used to determine the cost incurred by the tensor operation or scalar operation when running on a candidate processor, and the benefit function is used to determine the benefit incurred by the tensor operation or scalar operation when running on a candidate processor. Various implementations of the cost function and benefit function have been described in detail above and will not be repeated here.

[0052] To determine the cost function value and / or benefit function value, the disclosed solution includes a cost calculation unit 401 and / or a benefit calculation unit 402 within the computation allocation unit. In an exemplary specific operation, the cost calculation unit and the benefit calculation unit can utilize the function library 403 to calculate the cost or benefit value of a certain operation (as shown in the figure, Op[i]). In one embodiment, the function library disclosed herein can be implemented as a mapping set of operations and cost or benefit values; that is, for a certain operation (which may correspond to one or more operators), it has a cost or benefit value corresponding to running on a certain processor or processor core. These values ​​can be, for example, experimental values ​​or empirical values.

[0053] In one implementation scenario, the cost calculation unit or benefit calculation unit can use the corresponding function in the function library to calculate the corresponding cost and benefit values ​​based on parameters such as the data type or size involved in the operation. For example, in actual operation, when Op[i] is a scalar operation, the function library can provide a corresponding benefit value that can be obtained when running on, for example, the j-th processor core, based on the operator involved in the scalar operation, the data type and size when the operator performs the operation; when Op[i] is a tensor operation, a relatively large cost value caused by running on, for example, the j-th processor core can be obtained from the function library. Depending on the implementation method, the aforementioned j-th processor core can also be replaced by a general-purpose processor or an intelligent processor. Additionally, the aforementioned j-th processor core can also be one of the intelligent processor cores in a multi-core architecture computing system.

[0054] After obtaining the cost and benefit value of Op[i] running on the j-th processor core, at point 404, it can be determined whether the benefit value is greater than or equal to the first threshold ("Th1" in the figure). When the benefit value is greater than or equal to the first threshold, at point 405, it can be determined whether the cost value is less than or equal to the second threshold ("Th2" in the figure). When the cost value is less than or equal to the second threshold, Op[i] can be assigned to the j-th processor core for execution. As can be seen from the example given in the figure, in the comparison of the two thresholds, the operation can be assigned to the corresponding processor core (i.e., Core[j] in the figure) for execution only when the benefit value meets the minimum limit and the cost value does not exceed the maximum limit. Furthermore, through branch 407, the information of Op[i] being assigned to a specific processor core can also be recorded in the function library for consideration when generating the cost or benefit value of the subsequent operation, such as Op[i+1]. For example, when performing an allocation operation on Op[i+1], the cost calculation unit and the benefit calculation unit can calculate the cost and benefit values ​​obtained after fusing Op[i+1] and Op[i] using operators. Furthermore, when the obtained cost and benefit values ​​meet the threshold requirements, the subsequent operation Op[i+1] and the previous operation Op[i] can be allocated to the same processor core, that is, the processor core allocated to the previous operation Op[i].

[0055] As shown in branch 406 of the diagram, when the benefit value is less than the first threshold or the cost value is greater than the second threshold (i.e., neither the benefit nor the cost value reaches the preset expected value), the process shown in the diagram can return to the function library. Here, similar to branch 407, the state where Op[i] was not successfully assigned to Core[j] can be recorded in the function library. Next, the cost calculation unit and the benefit calculation unit can use the function library to calculate the cost and benefit value of Op[i] on the "j+1"th processor core (i.e., "Core[j+1]"), and compare them with the first and second thresholds mentioned above. Further, under the condition that the thresholds are met, Op[i] is assigned to the "j+1"th processor core. Similar to the scheme described above, in a scenario, Op[i] that was not successfully assigned to Core[j] can also be merged with Op[i-1]. Furthermore, the cost calculation unit and the benefit calculation unit can calculate the cost value and benefit value of the fused operation respectively, and when the cost value and benefit value of the fused operation meet the threshold requirements, Op[i] is assigned to the same processor core as Op[i-1].

[0056] The above is combined with the appendix Figure 4The computation allocation unit disclosed herein has been described. It is understood that the above description is merely exemplary and not restrictive, and those skilled in the art will conceive of appropriate modifications to the implementation of the computation allocation unit based on the teachings of this disclosure. For example, when the computation allocation unit of this disclosure utilizes… Figure 4 If the process shown still cannot determine which processor core Op[i] will be assigned to, it can choose to select the processor core with the lowest cost and / or highest benefit as the final processor core to be assigned.

[0057] Figure 5 This is a structural diagram illustrating a combined processing apparatus according to an embodiment of this disclosure. Figure 5 As shown, the combined processing device 500 includes a computing processing device 502, an interface device 504, other processing devices 506, and a storage device 508. Depending on the application scenario, the computing processing device may include one or more computing devices 510, which can be configured to perform various computing operations, such as the various operations involved in machine learning in the field of artificial intelligence.

[0058] In different embodiments, the computing processing apparatus disclosed herein can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Thus, the operation allocation scheme described above in conjunction with the accompanying drawings can reasonably distribute scalar or tensor operations across various processors. Similarly, one or more computing devices included within the computing processing apparatus can be implemented as an artificial intelligence processor core or a portion of the hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as artificial intelligence processor cores or portions of the hardware structure of artificial intelligence processor cores, the computing processing apparatus disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure.

[0059] In exemplary operation, the computing processing device disclosed herein can interact with other processing devices through an interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices disclosed herein may include one or more types of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and an artificial intelligence processor, both general-purpose and / or special-purpose processors. These processors may include, but are not limited to, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing processing device disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0060] In one or more embodiments, the other processing device may serve as an interface between the computing processing device disclosed herein (which may be specifically embodied in artificial intelligence, such as neural network operations) and external data and control, performing basic controls including but not limited to data transfer, starting and / or stopping the computing device. In another embodiment, the other processing device may also cooperate with the computing processing device to jointly complete computational tasks.

[0061] In one or more embodiments, the interface device can be used to transfer data and control instructions between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it to on-chip storage (or memory) of the computing processing device. Further, the computing processing device can obtain control instructions from other processing devices via the interface device and write them to on-chip control cache of the computing processing device. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit it to other processing devices. In some scenarios, the interface device can also be implemented as an application programming interface between the computing processing device and other processing devices, including, for example, a driver interface, to transfer various instructions and programs to be executed by the computing processing device between them.

[0062] Additionally or optionally, the combined processing apparatus disclosed herein may further include a storage device. As shown in the figures, the storage device is connected to both the computing processing device and the other processing device. In one or more embodiments, the storage device may be used to store data from the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage of the computing processing device or other processing device.

[0063] In some embodiments, this disclosure also discloses a chip (e.g. Figure 6 The chip shown is 602. In one implementation, this chip is a system-on-a-chip (SoC). This chip can be externally interfaced with devices such as... Figure 6 The external interface device 606 shown is connected to other related components. These related components may be, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. In some applications, the chip may integrate other processing units (e.g., video codecs) and / or interface modules (e.g., DRAM interfaces). In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip. In some embodiments, this disclosure also discloses a board that includes the aforementioned chip package structure. The following will be combined with… Figure 6 This board is described in detail.

[0064] Figure 6 This is a schematic diagram illustrating the structure of a board 600 according to an embodiment of this disclosure, which may include the intelligent processor architecture described in conjunction with the accompanying drawings. Figure 6 As shown, the board includes a storage device 604 for storing data, which includes one or more storage cells 610. This storage device can be connected and transmit data with the controller 608 and the aforementioned chip 602 via, for example, a bus. Furthermore, the board also includes an external interface device 606, configured for data relay or switching between the chip (or a chip in a chip package) and an external device 612 (e.g., a server or computer). For example, data to be processed can be transferred from the external device to the chip via the external interface device. Alternatively, the calculation results of the chip can be transmitted back to the external device via the external interface device. Depending on the application scenario, the external interface device can have different interface forms, such as a standard PCIe interface.

[0065] In one or more embodiments, the controller in the disclosed board can be configured to regulate the state of the chip. Therefore, in one application scenario, the controller may include a microcontroller (MCU) for regulating the operating state of the chip.

[0066] Based on the above combination Figure 5 and Figure 6 Based on the description, those skilled in the art will understand that this disclosure also discloses an electronic device or apparatus that may include one or more of the aforementioned boards, one or more of the aforementioned chips, and / or one or more of the aforementioned combined processing apparatus.

[0067] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0068] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0069] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0070] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0071] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0072] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0073] The foregoing can be better understood in accordance with the following terms:

[0074] Clause A1, A method for compiling a neural network model, comprising:

[0075] The neural network model is subjected to computational analysis to obtain the tensor and scalar operations to be performed when the neural network model is executed;

[0076] The performance metrics of the tensor and scalar operations when executed on multiple processors are evaluated; and

[0077] Based on the evaluation results, the tensor operations and scalar operations are assigned to the corresponding processors for execution.

[0078] Clause A2, the compilation method according to Clause A1, wherein evaluating the performance metrics of the tensor operations and scalar operations when executed on multiple processors includes:

[0079] The performance metrics of the tensor and scalar operations are evaluated using cost functions and / or reward functions.

[0080] The cost function is used to determine the cost of the tensor operation or scalar operation when it is run on the candidate processor, and the benefit function is used to determine the benefit of the tensor operation or scalar operation when it is run on the candidate processor.

[0081] Clause A3, the compilation method according to Clause A2, wherein the cost function includes data transfer cost and / or cumulative cost, wherein the data transfer cost is the cost incurred by transferring data during the execution of the tensor operation or scalar operation on the candidate processor, and the cumulative cost is obtained by accumulating the cost incurred by the continuous execution of the tensor operation or scalar operation on the candidate processor.

[0082] Clause A4. The compilation method according to Clause A3, wherein the benefit function includes speedup benefit and / or complexity benefit, wherein the speedup benefit is the computational efficiency benefit obtained by the tensor operation or scalar operation on the candidate processor, and the complexity benefit is the benefit obtained when the computational complexity of the tensor operation or scalar operation is reduced when the tensor operation or scalar operation is run on the candidate processor.

[0083] Clause A5, the compilation method according to any one of clauses A1-A4, wherein assigning the tensor operations and scalar operations to the appropriate processors for execution based on the evaluation result includes:

[0084] The cost function and the profit function are used to search a function library to obtain the cost value and profit value of the cost function and the profit function; and

[0085] The allocation of the tensor or scalar operation to one of the corresponding general-purpose processors or intelligent processors is determined based on the cost and benefit values.

[0086] Clause A6. The compilation method according to Clause A5, wherein a first threshold is set for the revenue function and a second threshold is set for the cost function, and wherein allocation to the appropriate processor includes allocating the tensor operation or scalar operation to one of the appropriate general-purpose processor or intelligent processor according to the following comparison:

[0087] Based on the comparison of the revenue value with the first threshold; and / or

[0088] Based on the comparison between the cost value and the second threshold.

[0089] Clause A7, the compilation method as described in Clause A6, wherein:

[0090] When the return value is greater than or equal to the first threshold, the corresponding tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors; or

[0091] When the cost value is less than or equal to the second threshold, the corresponding tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors.

[0092] Clause A8, the compilation method as described in Clause A6, wherein:

[0093] When the benefit value is greater than or equal to the first threshold and the cost value is less than or equal to the second threshold, the corresponding tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors.

[0094] Clause A9, the compilation method according to any one of Clauses A6-A8, wherein the intelligent processor includes a plurality of intelligent processor cores, and the compilation method includes assigning the tensor operation or scalar operation to one of the corresponding intelligent processor cores.

[0095] Clause A10, the compilation method according to Clause A1, wherein performing the computational analysis to obtain the tensor and scalar operations to be performed includes:

[0096] The neural network model is compiled according to the hardware configuration on which it is running, so as to obtain the tensor and scalar operations to be performed when executing the neural network model.

[0097] Clause A11, the compilation method according to Clause A10, wherein the compilation includes performing one or more of shape derivation, block partitioning and tensor optimization on the neural network model to obtain the tensor operations to be performed when executing the neural network model.

[0098] Clause A12, the compilation method according to claim A10, wherein the compilation further includes scalar optimization of scalar operations in the neural network model to obtain the scalar operations to be performed when executing the neural network model.

[0099] Clause A13, the compilation method according to Clause A11, wherein the compilation further includes scalar optimization of the scalar operations resulting from performing one or more of the shape derivation, block division and optimization processes.

[0100] Item A14, a compiler comprising:

[0101] The computational analysis unit is configured to perform computational analysis on the neural network model to obtain the tensor and scalar operations to be performed when the neural network model is executed.

[0102] The arithmetic allocation unit is configured as follows:

[0103] The performance metrics of the tensor and scalar operations when executed on multiple processors are evaluated; and

[0104] Based on the evaluation results, the tensor operations and scalar operations are assigned to the corresponding processors for execution.

[0105] Clause A15, the compiler according to Clause A14, wherein in evaluating the tensor and scalar operations, the operation allocation unit is configured to:

[0106] The tensor and scalar operations are evaluated using a cost function and / or a reward function.

[0107] The cost function is used to determine the cost of the tensor operation or scalar operation when it is run on the candidate processor, and the benefit function is used to determine the benefit of the tensor operation or scalar operation when it is run on the candidate processor.

[0108] Clause A16, the compiler as described in Clause A15, wherein the cost function includes data transfer cost and / or cumulative cost, wherein the data transfer cost is the cost incurred by transferring data during the execution of the tensor operation or scalar operation on the candidate processor, and the cumulative cost is obtained by accumulating the cost incurred by the continuous execution of the tensor operation or scalar operation on the candidate processor.

[0109] Clause A17. The compiler pursuant to Clause A16, wherein the benefit function includes speedup benefits and / or complexity benefits, wherein the speedup benefits are the benefits obtained by accelerating the execution of the tensor or scalar operation on the candidate processor, and the complexity benefits are the benefits obtained by reducing the computational complexity of the tensor or scalar operation when it is executed on the candidate processor.

[0110] Clause A18, the compiler as described in Clause A17, wherein the arithmetic allocation unit is further configured to:

[0111] Tensor operations and scalar operations are assigned to the appropriate general-purpose processors or intelligent processors.

[0112] Clause A19. The compiler according to Clause A18, wherein the arithmetic allocation unit is further configured to:

[0113] The cost function and the profit function are used to search a function library to obtain the cost value and profit value of the cost function and the profit function; and

[0114] The allocation of the tensor or scalar operation to one of the corresponding general-purpose processors or intelligent processors is determined based on the cost and benefit values.

[0115] Clause A20, the compiler according to Clause A19, wherein a first threshold is set for the benefit function and a second threshold is set for the cost function, and wherein the computation allocation unit is further configured to:

[0116] When the benefit value is greater than or equal to the first threshold and the cost value is less than or equal to the second threshold, the tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors.

[0117] Clause A21, the compiler according to Clause A20, wherein the intelligent processor includes a plurality of intelligent processor cores, and wherein the operation allocation unit is further configured to allocate the tensor operation or scalar operation to one of the corresponding intelligent processor cores.

[0118] Clause A22, the compiler described in Clause A14, further includes a compilation unit configured as follows:

[0119] The neural network model algorithm is compiled according to the hardware configuration for running the neural network model algorithm to obtain the tensor and scalar operations required to run the neural network algorithm.

[0120] Clause A23. The compiler as described in Clause A22, wherein the compilation unit includes one or more of a shape derivation unit, a block division unit, and a tensor optimization unit, which are respectively used to perform shape derivation, block division, and tensor optimization processing on the neural network model algorithm to obtain the tensor operations required to run the neural network model algorithm.

[0121] Clause A24. The compiler as described in Clause A23, wherein the compilation unit further includes a scalar optimization unit for performing scalar optimization on scalar operations in the neural network model algorithm to obtain the scalar operations required to run the neural network algorithm.

[0122] Clause A25, the compiler according to Clause A24, wherein the scalar optimization unit is further configured to perform scalar optimization on the scalar operations generated by the shape derivation unit, the block unit, and the tensor optimization unit performing shape derivation, block, and optimization processing.

[0123] Clause A26, the compiler described in any one of Clauses A14-A25, further includes:

[0124] A code generation unit is configured to generate code to be executed by a corresponding processor based on the allocation result of the operation allocation unit.

[0125] Clause A27, an apparatus for compiling a neural network model, comprising:

[0126] processor;

[0127] A memory that stores a program, which, when executed by a processor, causes the device to perform the compilation method described in any one of clauses A1-A13.

[0128] Clause A28. A computer-readable storage medium having stored thereon a program for compiling a neural network model, which, when executed by one or more processors, causes the compilation method described in any one of Clauses A1-A13 to be implemented.

[0129] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0130] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0131] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0132] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A compilation method for neural network models, comprising: The neural network model is subjected to computational analysis to obtain the tensor and scalar operations to be performed when the neural network model is executed; The performance metrics of the tensor and scalar operations are evaluated when they are executed on multiple processors. as well as Based on the evaluation results, the tensor operations and scalar operations are assigned to the corresponding processors for execution; The performance evaluation of tensor and scalar operations when executed on multiple processors includes: The performance metrics of the tensor and scalar operations are evaluated using cost functions and / or reward functions. The cost function is used to determine the cost of the tensor operation or scalar operation when it is run on the candidate processor, and the benefit function is used to determine the benefit of the tensor operation or scalar operation when it is run on the candidate processor.

2. The compilation method according to claim 1, wherein the cost function includes data transfer cost and / or cumulative cost, wherein the data transfer cost is the cost generated by transferring data when the tensor operation or scalar operation is run on the candidate processor, and the cumulative cost is obtained by accumulating the cost generated by the continuous operation of the tensor operation or scalar operation on the candidate processor.

3. The compilation method according to claim 2, wherein the benefit function includes acceleration benefit and / or complexity benefit, wherein the acceleration benefit is the computational efficiency benefit obtained by the tensor operation or scalar operation on the candidate processor, and the complexity benefit is the benefit obtained when the computational complexity is reduced when the tensor operation or scalar operation is run on the candidate processor.

4. The compilation method according to any one of claims 1-3, wherein assigning the tensor operations and scalar operations to corresponding processors for execution based on the evaluation result includes: Use the cost function and the benefit function to search the function library to obtain the cost value and benefit value of the cost function and the benefit function; as well as The allocation of the tensor or scalar operation to one of the corresponding general-purpose processors or intelligent processors is determined based on the cost and benefit values.

5. The compilation method of claim 4, wherein a first threshold is set for the benefit function and a second threshold is set for the cost function, and wherein allocation to the appropriate processor comprises allocating the tensor operation or scalar operation to one of the appropriate general-purpose processor or intelligent processor according to the following comparison: Based on the comparison of the revenue value with the first threshold; and / or Based on the comparison between the cost value and the second threshold.

6. The compilation method according to claim 5, wherein: When the profit value is greater than or equal to the first threshold, the corresponding tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors. or When the cost value is less than or equal to the second threshold, the corresponding tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors.

7. The compilation method according to claim 5, wherein: When the benefit value is greater than or equal to the first threshold and the cost value is less than or equal to the second threshold, the corresponding tensor operation or scalar operation is assigned to one of the corresponding general-purpose processors or intelligent processors.

8. The compilation method according to any one of claims 5-7, wherein the intelligent processor comprises a plurality of intelligent processor cores, and the compilation method comprises assigning the tensor operation or scalar operation to one of the corresponding intelligent processor cores.

9. The compilation method according to claim 1, wherein performing the operational analysis to obtain the tensor and scalar operations to be performed comprises: The neural network model is compiled according to the hardware configuration on which it is running, so as to obtain the tensor and scalar operations to be performed when executing the neural network model.

10. The compilation method according to claim 9, wherein the compilation includes performing one or more of shape derivation, block partitioning and tensor optimization on the neural network model to obtain the tensor operations to be performed when executing the neural network model.

11. The compilation method according to claim 9, wherein the compilation further includes scalar optimization of the scalar operations in the neural network model to obtain the scalar operations to be performed when executing the neural network model.

12. The compilation method of claim 10, wherein the compilation further comprises performing scalar optimization on the scalar operations generated by performing one or more of the shape derivation, block division and optimization processes.

13. A compiler, comprising: The computational analysis unit is configured to perform computational analysis on the neural network model to obtain the tensor and scalar operations to be performed when the neural network model is executed. The arithmetic allocation unit is configured as follows: Evaluating the performance metrics of the tensor operations and scalar operations when executed on multiple processors includes using a cost function and / or a benefit function to evaluate the performance metrics of the tensor operations and scalar operations, wherein the cost function is used to determine the cost incurred by the tensor operations or scalar operations when running on candidate processors, and the benefit function is used to determine the benefit incurred by the tensor operations or scalar operations when running on candidate processors; as well as Based on the evaluation results, the tensor operations and scalar operations are assigned to the corresponding processors for execution.

14. An apparatus for compiling a neural network model, comprising: processor; as well as A memory storing a program that, when executed by a processor, causes the device to perform the compilation method according to any one of claims 1-12.

15. A computer-readable storage medium having stored thereon a program for compiling a neural network model, which, when executed by one or more processors, causes the compilation method according to any one of claims 1-12 to be implemented.

Images