Npu data arrangement diagram optimization method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a NPU data arrangement graph optimization method and device, electronic equipment and storage medium. The method comprises: after obtaining original data arrangement with a dimension number P, in order to adapt to the dimension number Q supported by the operator in the neural network processor NPU, performing format conversion and adaptation on the dimension number of the original data arrangement according to a specified conversion scheme, to obtain a target data arrangement matched with the NPU, thereby solving the technical problem that the conversion mode of the data arrangement in the prior art is not adapted to the NPU, and by providing a data arrangement optimization scheme adapted to the NPU, the inference speed of the NPU can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and more specifically, the embodiments of the present invention relate to a graph optimization method, apparatus, electronic device and storage medium for NPU data layout. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] In related technologies, there are only suitable data arrangement formats for general neural network accelerators that are CPUs (central processing units) and GPUs (graphics processing units), usually NCHW or NHWC.

[0004] like Figure 1 As shown, in NCHW, N represents number, C represents channel, H represents height, and W represents width. NCHW actually represents [WHCN]. The first element is 000, the second element is along the w direction, i.e., 001, and so on, 002, 003, then along the H direction, i.e., 004, 005, 006, 007, and so on until 019, then along the C direction, i.e., 020, then 021, 022, all the way to 319, and then back along the N direction. NHWC is similar to NCHW, representing [CWHN]. The first element is 000, the second is along the C direction, i.e., 020, 040, 060, all the way to 300, then along the W direction, 001, 021, 041, 061...301, then after 303, along the H direction, i.e., 004, 024...304, finally reaching 319, which becomes the N direction.

[0005] In related technologies, optimizations have been made to achieve efficient computational processing on CPUs or GPUs, typically by pre-converting the data arrangement to an efficient layout supported by the CPU or GPU. However, the data arrangement of an NPU (neural-network processing unit) differs from that of CPUs and GPUs, often being five-dimensional (such as NCHWC) or even higher. Therefore, the data arrangement conversion methods used on CPUs or GPUs cannot be directly applied to NPUs. Summary of the Invention

[0006] In view of the technical problem that the data arrangement conversion method in related technologies is not compatible with NPU, in this context, embodiments of the present invention aim to provide a graph optimization method, apparatus, electronic device and storage medium for NPU data arrangement to solve the above problems.

[0007] In a first aspect of the present invention, a graph optimization method for NPU data layout is provided, comprising: obtaining the original data layout of operators in a deep learning model, wherein the deep learning model runs on a neural network processor (NPU), the original data layout adopts a first data layout format, the dimension of the first data layout format is P, and P is a positive integer; converting the original data layout according to a specified conversion scheme to obtain a target data layout adapted to the neural network processor, wherein the target data layout adopts a second data layout format, the dimension of the second data layout format is Q, and Q is a positive integer greater than P, and the specified conversion scheme is used to convert the dimension of the data layout format from P to Q and eliminate the influence of the conversion process on the calculation results.

[0008] In one embodiment of the present invention, transforming the original data arrangement according to a specified transformation scheme to obtain a target data arrangement adapted to a neural network processor (NPU) includes: aligning the C dimension of the original data arrangement to S to obtain an intermediate data arrangement, wherein S is 2. n , where n is a preset positive integer; the intermediate data arrangement is converted from the first data arrangement format to the second data arrangement format to obtain the target data arrangement.

[0009] In another embodiment of the present invention, after converting the original data arrangement according to the specified conversion scheme to obtain the target data arrangement adapted to the neural network processor NPU, the method further includes: classifying the operators in the deep learning model into multiple types according to whether the calculation results of the operators in the deep learning model are affected by the conversion; configuring first attribute information according to each operator of the type belonging to the deep learning model so that the calculation results of the operators are not affected by the conversion.

[0010] In another embodiment of the present invention, operators in a deep learning model are classified into several types according to whether the calculation results of operators in the deep learning model are affected by the transformation. These include: operators in a deep learning model are classified into three categories according to whether the calculation results of operators in the deep learning model are affected by the transformation: a first type of operator, a second type of operator, and a third type of operator. The first type of operator is an operator that does not need to operate on the C dimension in the data arrangement and whose result is not affected; the second type of operator is an operator that needs to operate on the C dimension in the data arrangement and whose result is affected; and the third type of operator is an operator used to process a specified data arrangement format and whose result is affected.

[0011] In another embodiment of the present invention, configuring first attribute information for each operator belonging to the type of deep learning model to ensure that the calculation result of the operator is not affected by the transformation includes: configuring the first attribute information of the i-th operator in the deep learning model, where i is a positive integer less than or equal to the total number of operators K in the deep learning model; when the i-th operator belongs to the first type of operator, setting the pad_info of the output tensor of the i-th operator to be the same as the pad_info of the input tensor, where pad_info is used by the next operator to determine whether to perform data restoration; when the i-th operator belongs to the second type of operator, setting the pad_info of the output tensor of the i-th operator to []; when the i-th operator belongs to the third type of operator, setting the pad_info of the output tensor of the i-th operator to [C, C2]. n [C, C2] n In the figure, C represents the original length of the tensor, C2 n This indicates the length after conversion; if the current value of i is less than the total number of operators K, increment the value of i by 1, and continue to execute the step of configuring the attribute information of the i-th operator in the deep learning model according to the changed value of i; if the current value of i is equal to the total number of operators K, end the configuration of the attribute information of the operators in the deep learning model.

[0012] In another embodiment of the present invention, the intermediate data arrangement is converted from a first data arrangement format to a second data arrangement format to obtain a target data arrangement, including: parsing the intermediate data arrangement according to the first data arrangement format to obtain data for each dimension of the intermediate data arrangement; sorting the data for each dimension of the intermediate data arrangement according to the second data arrangement format to obtain the target data arrangement, wherein the data for dimension C in the intermediate data arrangement is equal to the product of the data for each dimension C in the target data arrangement.

[0013] In another embodiment of the present invention, before converting the intermediate data arrangement from a first data arrangement format to a second data arrangement format to obtain the target data arrangement, the method further includes: configuring second attribute information for each operator of the deep learning model to reduce the addition of conversion nodes for the input or output data arrangement.

[0014] In another embodiment of the present invention, configuring second attribute information for each operator of the deep learning model to reduce the addition of transformation nodes for the data arrangement of input or output includes: establishing a mapping table op_map and a tensor mapping table tensor_map for each operator in the deep learning model, wherein the mapping table includes the mapping relationship between each position in the input tensor of the operator and each input position of the neural network processor, and the tensor mapping table is used to indicate the mapping relationship between the first data arrangement format and the second data arrangement format; and selecting at least one from inserting transformation nodes, modifying the second attribute information, and keeping the second attribute information unchanged, based on the characteristics of the operators in the deep learning model, the mapping table, and the tensor mapping table tensor_map.

[0015] In a second aspect of the present invention, a graph optimization apparatus for NPU data layout is provided, comprising: an acquisition unit for acquiring the original data layout of operators in a deep learning model, wherein the deep learning model runs on a neural network processor (NPU), the original data layout adopts a first data layout format, the dimension of the first data layout format is P, and P is a positive integer; and an optimization unit for converting the original data layout according to a specified conversion scheme to obtain a target data layout adapted to the neural network processor, wherein the target data layout adopts a second data layout format, the dimension of the second data layout format is Q, Q is a positive integer greater than P, and the specified conversion scheme is used to convert the dimension of the data layout format from P to Q and eliminate the influence of the conversion process on the calculation results.

[0016] In one embodiment of the present invention, the optimization unit is further configured to: align dimension C in the original data arrangement to S to obtain an intermediate data arrangement, wherein S is 2. n , where n is a preset positive integer; the intermediate data arrangement is converted from the first data arrangement format to the second data arrangement format to obtain the target data arrangement.

[0017] In another embodiment of the present invention, the optimization unit is further configured to: after converting the original data arrangement according to the specified conversion scheme to obtain the target data arrangement adapted to the neural network processor NPU, classify the operators in the deep learning model into multiple types according to whether the calculation results of the operators in the deep learning model are affected by the conversion; configure first attribute information according to each operator belonging to the deep learning model so that the calculation results of the operators are not affected by the conversion.

[0018] In another embodiment of the present invention, the optimization unit is further configured to: classify the operators in the deep learning model into three categories according to whether the calculation results of the operators in the deep learning model are affected by the transformation: a first category of operators, a second category of operators, and a third category of operators, wherein: the first category of operators are operators that do not need to operate on the C dimension in the data arrangement and whose results are not affected; the second category of operators are operators that need to operate on the C dimension in the data arrangement and whose results are affected; and the third category of operators are operators used to process a specified data arrangement format and whose results are affected.

[0019] In another embodiment of the present invention, the optimization unit is further configured to: configure the first attribute information of the i-th operator in the deep learning model, where i is a positive integer less than or equal to the total number of operators K in the deep learning model; when the i-th operator belongs to the first type of operator, set the pad_info of the output tensor of the i-th operator to be the same as the pad_info of the input tensor, and pad_info is used by the next operator to determine whether to perform data restoration; when the i-th operator belongs to the second type of operator, set the pad_info of the output tensor of the i-th operator to []; when the i-th operator belongs to the third type of operator, set the pad_info of the output tensor of the i-th operator to [C, C2]. n [C, C2] n In the figure, C represents the original length of the tensor, C2 n This indicates the length after conversion; if the current value of i is less than the total number of operators K, increment the value of i by 1, and continue to execute the step of configuring the attribute information of the i-th operator in the deep learning model according to the changed value of i; if the current value of i is equal to the total number of operators K, end the configuration of the attribute information of the operators in the deep learning model.

[0020] In another embodiment of the present invention, the optimization unit is further configured to: parse the intermediate data arrangement according to the first data arrangement format to obtain the data of each dimension in the intermediate data arrangement; sort the data of each dimension in the intermediate data arrangement according to the second data arrangement format to obtain the target data arrangement, wherein the data of dimension C in the intermediate data arrangement is equal to the product of the data of each dimension C in the target data arrangement.

[0021] In another embodiment of the present invention, the optimization unit is further configured to: configure second attribute information for each operator of the deep learning model before converting the intermediate data arrangement from the first data arrangement format to the second data arrangement format to obtain the target data arrangement, so as to reduce the addition of conversion nodes for the input or output data arrangement.

[0022] In another embodiment of the present invention, the optimization unit is further configured to: establish a mapping table op_map and a tensor mapping table tensor_map for each operator in the deep learning model, wherein the mapping table includes the mapping relationship between each position in the input tensor of the operator and each input position of the neural network processor, and the tensor mapping table is used to indicate the mapping relationship between the first data layout format and the second data layout format; and select at least one from inserting a transformer node, modifying the second attribute information, and keeping the second attribute information unchanged to perform the operation based on the characteristics of the operators in the deep learning model, the mapping table, and the tensor mapping table tensor_map.

[0023] In a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps described in the method embodiments above.

[0024] In a fourth aspect of the present invention, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute the steps described in the apparatus embodiments at runtime.

[0025] In related technical solutions, deep learning compilers typically convert existing deep learning framework models such as PyTorch, ONNX, and TensorFlow into specific intermediate models (IRs), followed by data arrangement optimization passes, such as converting NCHW to a data format that is efficiently supported by CPUs or GPUs. However, these conversions are usually limited to the same dimensionality. In NPUs, however, not only is data arrangement conversion required, but also transformations from 4 to 5 dimensions or even higher. The technical solution of this application, after obtaining the original data arrangement with dimension P, performs format conversion and adaptation according to a specified conversion scheme to match the dimension Q supported by operators in the neural network processor (NPU), resulting in a target data arrangement that matches the NPU. This solves the technical problem of incompatibility between existing data arrangement conversion methods and NPUs. By providing a data arrangement optimization scheme adapted to the NPU, the inference speed of the NPU can be improved. Attached Figure Description

[0026] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0027] Figure 1 The data layout format according to an embodiment of the present invention is illustrated schematically;

[0028] Figure 2 An application scenario according to an embodiment of the present invention is illustrated schematically;

[0029] Figure 3 A graph optimization method for NPU data layout according to an embodiment of the present invention is illustrated schematically.

[0030] Figure 4 The diagram illustrates a model conversion method according to yet another embodiment of the present invention;

[0031] Figure 5 A graph optimization method according to yet another embodiment of the present invention is illustrated schematically;

[0032] Figure 6 This schematically illustrates a graph optimization apparatus for NPU data layout according to an exemplary embodiment of the present invention;

[0033] Figure 7 A schematic diagram of the structure of an electronic device according to another embodiment of the present invention is shown.

[0034] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0035] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art. Invention Overview

[0037] Through analysis of relevant technologies, the inventors realized that with the widespread application of deep learning in image, speech recognition, and natural language translation, deep learning algorithms are also being implemented on various computing platforms (such as CPU, GPU, NPU, etc.).

[0038] During the deployment process, the data arrangement will be transformed from ONNX (Open Neural Network Exchange, an open file format designed for machine learning, which can convert models from any training framework, such as TensorFlow, PyTorch, OneFlow, Paddle, etc., to ONNX for storage after training) and PyTorch (a Python-based sustainable computing package with powerful GPU-accelerated tensor computation, such as NumPy, and deep neural networks with automatic differentiation systems) to deployable models. The data arrangement transformation is a tail transformation of equal length, that is, from NCHW to NHWC, or from NHWC to NCHW.

[0039] For data layout transformation methods on CPUs or GPUs, some open-source frameworks, such as MNN (a lightweight, high-performance inference engine that loads models and performs inference on devices) and NCNN (a high-performance neural network feedforward computation framework optimized for mobile devices), offer feasible solutions. These solutions involve inserting data layout transformation nodes at the beginning and end of the model, while modifying the attribute values ​​of the intermediate computation nodes based on their characteristics.

[0040] In general-purpose CPUs or GPUs, the data layout format for convolution is typically NCHW or NHCW. However, the NPU, as a dedicated accelerator at the edge, is widely used in scenarios with high real-time requirements, such as intelligent learning products and transportation, due to its low power consumption and low latency. As a dedicated neural network accelerator, the NPU needs to efficiently process convolution operations and matrix multiplication calculations. Therefore, the NPU has specific dedicated hardware circuitry, which requires a compatible data layout format. Consequently, a deep learning compiler is needed to provide a compatible deep learning model.

[0041] To fully utilize bandwidth and Processing Elements (PEs, typically including a multiply-accumulate unit, a few registers, and a small amount of control logic), NPUs usually employ customized data layout formats such as NHCWC16 (C16 represents a group of 16 channels, where C*C16 equals C in NCHW) and NCHC4WC4 (C4 represents a group of 4 channels, where C*C4*C4 equals C in NCHW). To reduce the number of computation nodes in the model and to handle some data transformations, graph optimization capabilities are required for deep learning. The solution in this application optimizes and adapts the NPU's data layout using a graph.

[0042] 1) In the general scheme, the transformation only applies to the calculation of convolution (conv) and tensor multiplication (matmul, general matrix multiplication (gemm)). It changes the weights for this type of calculation, but the data arrangement of other calculation nodes is still NCHW. This scheme extends the transformation of data arrangement from the same dimension to different dimensions. For example, the dimension changes from 4-dimensional to 5-dimensional, or from 3-dimensional to 4-dimensional, etc. In addition, C in the original data arrangement (such as NCHW) is not necessarily a multiple of 16 (it can be any other power of 2), but the transformed dimension (such as NCHWC) must be a multiple of 16 (or any other power of 2).

[0043] 2) This solution addresses the problem of transforming data arrangement into forward propagation in a neural network (where each node's computation depends on the result of the previous node, and the transformation of data arrangement also depends on the computation graph, meaning the data arrangement for the next computation depends on the data arrangement of the previous node). This solution is independent of the type of computation node (this refers to computation; different nodes have different computation methods, but for data arrangement, the same formula can be used to express it). Furthermore, it minimizes the insertion of redundant nodes. Based on the current node's computation type, attributes, and the data arrangement of the input data, only the attributes of the current node need to be adjusted, without needing to insert other operators to restore the original data arrangement to suit the current node's computation.

[0044] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0045] Application Scenarios Overview

[0046] First, refer to Figure 2 , Figure 2 The application scenario according to an embodiment of the present invention is illustrated schematically. First, refer to... Figure 2 This illustrates an application scenario in which embodiments of the present invention can be implemented. Figure 2 The scenario shown includes terminal 200 and server 100. Terminal 200 can be a mobile terminal, such as a mobile phone, tablet, laptop, personal digital assistant, or desktop computer. Server 100 can be a software operator's server. Terminal 200 and server 100 can communicate via, for example, mobile internet.

[0047] When graph optimization of the NPU data layout is required, the server 100 can perform the following steps: obtain the original data layout of the operators in the deep learning model. The deep learning model runs on the neural network processor (NPU). The original data layout adopts a first data layout format, and the dimension of the first data layout format is P, where P is a positive integer. The original data layout is transformed according to a specified transformation scheme to obtain a target data layout adapted to the neural network processor. The target data layout adopts a second data layout format, and the dimension of the second data layout format is Q, where Q is a positive integer greater than P. The specified transformation scheme is used to convert the dimension of the data layout format from P to Q and eliminate the influence of the transformation process on the calculation results.

[0048] Of course, the above steps can also be performed by terminal 200, or by server 100 and terminal 200 together (such as one of server 100 and terminal 200 performing a part of the steps, while the other performs the remaining steps).

[0049] Exemplary methods

[0050] The following is combined with Figure 2 Application scenarios, refer to Figure 3 The graph optimization method for NPU data layout according to an exemplary embodiment of the present invention is described.

[0051] It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0052] For example, see Figure 3 The diagram shown is a flowchart of a graph optimization method for NPU data layout according to an embodiment of the present invention. Figure 3 As shown, the graph optimization method for NPU data layout may include the following steps:

[0053] Step S1: Obtain the original data arrangement of operators in the deep learning model. The deep learning model runs on the neural network processor (NPU).

[0054] In this embodiment, the original data layout is the data layout before optimization. It adopts a first data layout format with a dimension number of positive integer P. The first data layout format can be NCHW, NHWC, etc. For example, when it is NCHW or NHWC, the corresponding value of P is 8.

[0055] Step S2: Transform the original data layout according to the specified transformation scheme to obtain the target data layout adapted to the neural network processor. The target data layout is the optimized data layout, which adopts the second data layout format with a dimension of a positive integer Q.

[0056] In this embodiment, the specified conversion scheme has two main functions: first, format conversion, which converts the number of dimensions of the data layout format from P to Q and adaptively adjusts the order of each dimension; second, eliminating the impact of the conversion process on the calculation results, such as adjusting the attributes of the nodes according to the calculation type, attributes of the nodes in the operator and the data layout of the input data to adapt to the calculation of the nodes (i.e., the accuracy of the calculation results will not be affected by the above conversion).

[0057] In relevant technical solutions, the model conversion process of a deep learning compiler is as follows: Figure 4 As shown, existing deep learning framework models such as PyTorch, ONNX, and TensorFlow are typically converted into specific intermediate models (IR), and then data arrangement optimization passes are added (which can be understood as converting a graph from A to B; this conversion process is usually called a pass, and operations such as constant folding and elimination of dead nodes can be achieved through passes). For example, NCHW is converted to a data format that is efficiently supported by CPUs or GPUs (such as GPUs efficiently supporting NHWC). However, these conversions are usually conversions of the same dimension. In NPUs, in addition to data arrangement conversions, transformations from 4 to 5 dimensions or even higher are required. Through the steps described in this application, after obtaining the original data arrangement with a dimension of P, in order to adapt to the dimension Q supported by the operators in the neural network processor NPU, the dimension of the original data arrangement is converted and adapted according to a specified conversion scheme to obtain a target data arrangement that matches the NPU. This solves the technical problem that the data arrangement conversion method in the prior art is not compatible with the NPU. By providing a data arrangement optimization scheme that is compatible with the NPU, the inference speed of the NPU can be improved.

[0058] In this embodiment of the invention, a general data format stream conversion method is provided, which effectively reduces the increase in computing nodes caused by data arrangement conversion, thereby increasing the inference speed of the model. As an optional embodiment, the technical solution of this application is further described in detail below with reference to specific embodiments:

[0059] Step S1: Obtain the original data arrangement of the operators in the deep learning model so as to convert it into a target data arrangement that matches the operators.

[0060] Here, we take the original ONNX data stream as NCHW, which needs to be converted to the NPU-adapted data stream NHCWC16 as an example. At this time, the first data layout format is NCHW with 4 dimensions, and the second data layout format is NHCWC16 with 5 data dimensions.

[0061] Of course, the first and second data layout formats here can also be other formats. This embodiment is only used for illustrative purposes. It is only necessary to satisfy that the number of dimensions of the second data layout format is greater than the number of dimensions of the first data layout format.

[0062] Step S2: Transform the original data layout according to the specified transformation scheme to obtain the target data layout adapted to the neural network processor.

[0063] The conversion between NCHW and NHCWC16 in this application embodiment can be abstracted as a conversion from data layout A to data layout B, and is limited to the fact that the floor function of dimension C in data layout A, Ceiling(C / align)*align, is equal to the product of all dimensions containing C in data layout B (i.e., C2). n (e.g., C16), where align is the minimum number of computation channels supported by the hardware, and what needs to be transformed here is tensor_map. tensor_map is a mapping relationship, which can be represented as tensor_map = [A_index, reflact], where A_index = the index of each element in A, reflact = the position of each element in B in A, and all positions in B containing C are consistent with the positions of C in A.

[0064] The above-described data layout conversion from NCHW to NHCWC16 generally imposes constraints on the convolution operator. It mainly consists of the following steps S21-S22:

[0065] Step S21: Align dimension C in the original data arrangement to S (i.e., 2). n (where n is a preset positive integer), to obtain the intermediate data arrangement.

[0066] Continuing with the previous example, taking the conversion of NCHW to NHCWC16 as an example, in step S21, the data arrangement of dimension C in NCHW can be aligned to an integer multiple of 16, that is, converting NCHW to NC. align16 HW (i.e., intermediate data layout).

[0067] Deep learning inference is a forward computation process. The computation of the current node (or operator) depends on the computation result of the previous node. There are about 40 types of operators commonly used in deep learning. Operators can be classified into various types according to whether their computation results are affected by transformations. Then, corresponding attribute information can be configured for operators according to their type so that the computation results of operators are not affected by transformations.

[0068] For example, in convolution operators, aligning the input to 16-bit will not affect the calculation result. Therefore, there is no need to insert pad or depad operators (the pad operator is used to pad and expand the original data according to a specified width, which can be to fill the beginning, end, or a specified position with "0"; the depad operator is the opposite of the pad operator and is used to delete and reduce the data). However, for softmax and reduce operators, if aligning the C-dimensional data to 16-bit will affect the calculation result, the depad operator needs to be inserted to avoid the impact of aligning the C-dimensional data to 16-bit. If the operation is performed on other dimensions, the depad operator does not need to be inserted.

[0069] As can be seen, for each operator, whether to insert a pad or depad node depends on the calculation result of the previous node and the attributes of the current calculation node. The specific implementation is as follows:

[0070] First, operators are divided into three categories based on whether the calculation results are affected.

[0071] The first type of operator is the operator whose calculation result is not affected (the data arrangement is not limited to NCHWC16), such as the conv operator, the relu operator, and reduce-type operators, softmax operators, and split operators that do not operate on the C dimension.

[0072] The second type of operator is the node whose calculation result is affected (the data arrangement of the operator is not limited to NCHWC16), such as the reduce operator, softmax operator and adds operator that need to operate on the C dimension;

[0073] The third type of operator is the operator whose data layout must be NCHWC16 (i.e., the specified data layout format) and whose calculation results are affected, such as the conv operator and the pool operator.

[0074] Secondly, traverse the directed acyclic topology of the deep learning model, and the pad_info of the network's input tensor (which is the basic unit of operation and represents a multidimensional matrix) is empty.

[0075] Each operator in the deep learning model can be iterated over to configure its first attribute information (i.e., pad_info, used by the next operator to determine whether to perform data restoration to eliminate the effects of transformation): if it is a first-type operator, its output tensor's pad_info is set to be the same as the input tensor's pad_info; if it is a second-type operator, its output tensor's pad_info is set to []; if it is a third-type operator, its output tensor's pad_info is set to [C, C2]. n [C, C2] nIn the figure, C represents the original length of the tensor, C2 n This indicates the length after conversion.

[0076] If the current node is a conv operator, and the input C is not 16-aligned, then a pad operator is added (the pad operator's function is to transform C into a multiple of 16). align16 =Ceiling(C / 16)*16), if the output C is not 16-aligned (whether the output is 16-aligned depends on the C dimension in the weights of the conv operator, which is one of the characteristics of the network itself), then the pad_info of the output tensor is set to [C,C align16 The purpose of pad_info is to calculate the padding method of the current data. If it encounters a computation node that has an impact and the data is the input of this computation node, it is necessary to perform a restoration operation, that is, insert the depad operator or the gather operator (usually used to take a certain dimension of the shape) to perform the corresponding operation.

[0077] If the current node is an operator with no effect, then: out_tensor_pad_info = input_tensor_pad_info, out_tensor_pad_info is the pad_info of the output tensor, and input_tensor_pad_info is the pad_info of the input tensor.

[0078] If the current operator is an influential operator, then the output of the operator will be operated on using the depad or gather operator. In this case, out_tensor_pad_info = [dim, dim].

[0079] It should be noted that the standard out_tensor_pad_info = [dim,pad_dim], where dim is the size of the original data in out_tensor, and pad_dim is the size of the data after padding. The calculation method of this value depends on whether padding has occurred in the current tensor or the input tesnor of this node.

[0080] like Figure 5 As shown, all operators are iterated through in the manner described above.

[0081] Step S22: Convert the intermediate data layout from the first data layout format to the second data layout format to obtain the target data layout.

[0082] In embodiments of this application, a second attribute information (e.g., axis information of a node) can be configured for each operator in the deep learning model to reduce the addition of transformation nodes for the data arrangement of input or output: an operator mapping table op_map (i.e., the mapping relationship between each position in the input tensor of the operator and each input position of the neural network processor) and a tensor mapping table tensor_map (i.e., the mapping relationship between the first data arrangement format and the second data arrangement format) are established, and then, based on the characteristics of the operators in the deep learning model, the mapping table, and the tensor mapping table tensor_map, at least one of the following can be selected to be executed: inserting transformation nodes, modifying the second attribute information, and keeping the second attribute information unchanged.

[0083] Following the previous example, step S22 can be understood as converting the NC... align16 HW is converted to NHCWC16, where C*C16 = C align16 .

[0084] Related technologies add transform nodes before and after conv, which increases the number of computation nodes. However, this application is based on the computational characteristics of neural networks. It performs forward propagation on the data arrangement of tensors (forward propagation uses the output of the previous layer as the input of the next layer and calculates the output of the next layer until the output layer is reached). Based on the characteristics of specific operators (including: the data arrangement must be NCHWC16 nodes, such as conv and pool operators; whether the computation results of computation nodes are affected), the properties of operators are modified or nodes are inserted.

[0085] The aforementioned attribute modifications typically refer to altering the axis information of the current node. The aim is to reduce the need for additional nodes that transform the input or output data layout. For example, in a `concat` node, if the original data layout was NCHW and the axis information `axis=1` (meaning it operates on the c-axis), but the current data layout is NHCWC16, only `axis` needs to be modified to 2. Since `concat` is not a computation node and does not perform calculations, only the attribute needs to be modified.

[0086] The aforementioned node insertion is for nodes whose computational results are affected. The data format needs to be restored to its original state, or a depad or gather operator must be inserted. By inserting a computation node on the graph while maintaining the original graph's topology, the hardware's data flow requirements can be adapted. For example, in the conv convolution operator, a node transformation involving data arrangement (NCHW->NHCWC16) is inserted.

[0087] The specific implementation process is as follows: Steps S221-S223:

[0088] Step S221: Establish the mapping table op_map for each operator.

[0089] Set the onnx_layout (layout refers to the layout in memory) and npu_layout of the input tensor of the network to [0,1,2...n-1], where n is the shape length of the tensor (shape refers to the shape of the tensor).

[0090] Based on the characteristics of each operator, its op_map is calculated, and the layout of the output tensor is inferred. For operators that do not change the tensor length, such as conv and relu operators, out_onnx_layout is calculated according to the operator type (here, operator type refers to commonly used operator types in neural networks). (This is determined by in_onnx_layout and the operator type, specifically whether the operator has the perm attribute. If it does, in_onnx_layout is transposed to obtain out_onnx_layout; otherwise, out_onnx_layout = in_onnx_layout). op_map = [in_onnx_layout, out_onnx_layout]. For operators that change the tensor length, out_onnx_layout is augmented or deleting dimensions (here, augmentation or deletion refers to augmentation or deletion of data dimensions, which is determined based on the type of computation node and the current value of the computation node, and can be determined in advance according to the actual application). The corresponding dimension index is then calculated for op_map. For example, the reshape operator, if the input shape is [3,2,6] and the output is [3,2,2,3], then op_map = [[0,1,2],[0,1,2,2]]. If in_onnx_layout = [0,1,2], then out_onnx_layout = [0,1,2,3].

[0091] The purpose of the above dimension index calculation is to record the index relationship of each dimension between the input and output. The shape mentioned above represents the size of each dimension of a tensor. For example, the input shape of the input tensor is [dim_a, dim_b, dim_c], and the output shape is [dim_a, dim_b1, dim_b2, dim_c], where dim_b = dim_b1 * dim_b2. In_onnx_layout is the subscript of the shape of the input tensor, i.e., in_onnx_layout = [0, 1, 2], and similarly out_onnx_layout = [0, 1, 2, 3].

[0092] One purpose of the above mapping relationship `op_map` is to record the correspondence between each `dim` of the input shape and the output shape. For example, `dim_b = dim_b1 * dim_b2`, `dim_a = dim_a`, `dim_c = dim_c`, `op_map = [a, reflact_a]`, where `a` is `in_onnx_la` and `reflact_a` is obtained. Specifically, it iterates through each `dim` of the input shape, sequentially finding in the output shape that is equal to the current `dim` or that the product of several consecutive `dim`s in the output shape is equal to the current `dim`. If they are equal, then `reflact_a = [dim]`. If there are n equal ones, then `reflact_a = [dim, dim, dim]`, where `reflact_a` has n elements, until each `dim` of the input shape has been traversed.

[0093] Step S222: Initialize tensor_map.

[0094] The input tensor_map of the conv operator is [[0,1,2,3],[0,2,1,3,1]], and the other tensor_maps are []. The tensor_map refers to the transformation method from the original data stream to the NPU data stream, that is, from (N,C,H,W) to (N,H,C,W,C16), corresponding to (0,1,2,3)->(0,2,1,3,1).

[0095] Step S223: Based on each operator characteristic, op_map, and tensor_map, insert a transform node or modify the operator attributes, or leave them unchanged.

[0096] Calculate infer_npu_layout (equivalent to the inferred value of npu_layout) = tensor_map[in_onnx_layout]. If infer_in_npu_layout is not equal to in_npu_layout (i.e., the actual value of npu_layout), insert a transform and set in_npu_layout to infer_in_npu_layout. If infer_in_npu_layout is equal to in_npu_layout and tensor_map is empty, do nothing. If tensor_map is not empty, if op_map[0] = op_map[1] (0 and 1 in the formula represent array indices), calculate new_axis = tensor_map[axis]. If op_map[0] is not equal to op_map[1], a transform node needs to be inserted to restore in_npu_layout to in_onnx_layout, and input tensor_map = []. Finally, calculate out_npu_layout = op_map[in_npu_layout]. Then set out_tensor_map to in_tensor_map.

[0097] Exemplary device

[0098] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 6 An exemplary embodiment of the NPU data layout graph optimization apparatus of the present invention will be described.

[0099] The implementation of the NPU data layout graph optimization device can refer to the implementation of the above method, and repeated details will not be elaborated further. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0100] Figure 6 A graph optimization apparatus for NPU data layout according to another embodiment of the present invention is illustrated schematically, the apparatus comprising:

[0101] The acquisition unit 601 is used to acquire the original data arrangement of the operators in the deep learning model, wherein the deep learning model runs on the neural network processor NPU, and the original data arrangement adopts a first data arrangement format, the number of dimensions of the first data arrangement format is P, and P is a positive integer;

[0102] The optimization unit 603 is used to transform the original data arrangement according to a specified transformation scheme to obtain a target data arrangement that is compatible with the neural network processor. The target data arrangement adopts a second data arrangement format, and the number of dimensions of the second data arrangement format is Q, where Q is a positive integer greater than P. The specified transformation scheme is used to convert the number of dimensions of the data arrangement format from P to Q and eliminate the influence of the transformation process on the calculation results.

[0103] Optionally, the optimization unit is also used to: align dimension C in the original data layout to S to obtain an intermediate data layout, where S is 2. n , where n is a preset positive integer; the intermediate data arrangement is converted from the first data arrangement format to the second data arrangement format to obtain the target data arrangement.

[0104] Optionally, the optimization unit is also used to: after transforming the original data arrangement according to the specified transformation scheme to obtain the target data arrangement adapted to the neural network processor NPU, classify the operators in the deep learning model into multiple types according to whether the calculation results of the operators in the deep learning model are affected by the transformation; configure first attribute information for each operator in the deep learning model according to its type so that the calculation results of the operators are not affected by the transformation.

[0105] Optionally, the optimization unit is further configured to: classify the operators in the deep learning model into three categories according to whether the computation results of the operators in the deep learning model are affected by the transformation: Category I operators, Category II operators, and Category III operators, wherein: Category I operators are operators that do not need to operate on the C dimension in the data arrangement and whose results are not affected; Category II operators are operators that need to operate on the C dimension in the data arrangement and whose results are affected; and Category III operators are operators used to process a specified data arrangement format and whose results are affected.

[0106] Optionally, the optimization unit is further configured to: configure the first attribute information of the i-th operator in the deep learning model, where i is a positive integer less than or equal to the total number of operators K in the deep learning model; if the i-th operator belongs to the first type of operator, set the pad_info of the output tensor of the i-th operator to be the same as the pad_info of the input tensor, and pad_info is used by the next operator to determine whether to perform data restoration; if the i-th operator belongs to the second type of operator, set the pad_info of the output tensor of the i-th operator to []; if the i-th operator belongs to the third type of operator, set the pad_info of the output tensor of the i-th operator to [C, C2]. n [C, C2] n In the figure, C represents the original length of the tensor, C2 nThis indicates the length after conversion; if the current value of i is less than the total number of operators K, increment the value of i by 1, and continue to execute the step of configuring the attribute information of the i-th operator in the deep learning model according to the changed value of i; if the current value of i is equal to the total number of operators K, end the configuration of the attribute information of the operators in the deep learning model.

[0107] Optionally, the optimization unit is further configured to: parse the intermediate data arrangement according to the first data arrangement format to obtain the data of each dimension in the intermediate data arrangement; sort the data of each dimension in the intermediate data arrangement according to the second data arrangement format to obtain the target data arrangement, wherein the data of dimension C in the intermediate data arrangement is equal to the product of the data of each dimension C in the target data arrangement.

[0108] Optionally, the optimization unit is also used to: configure second attribute information for each operator of the deep learning model before converting the intermediate data layout from the first data layout format to the second data layout format to obtain the target data layout, so as to reduce the addition of transformation nodes to the input or output data layout.

[0109] Optionally, the optimization unit is further configured to: establish a mapping table op_map and a tensor mapping table tensor_map for each operator in the deep learning model, wherein the mapping table includes the mapping relationship between each position in the input tensor of the operator and each input position of the neural network processor, and the tensor mapping table is used to indicate the mapping relationship between the first data layout format and the second data layout format; and select at least one from inserting a transformer node, modifying the second attribute information, and keeping the second attribute information unchanged to perform the operation based on the characteristics of the operators in the deep learning model, the mapping table, and the tensor mapping table tensor_map.

[0110] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0111] Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0112] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0113] S1, obtain the original data arrangement of the operators in the deep learning model, wherein the deep learning model runs on the neural network processor NPU, the original data arrangement adopts the first data arrangement format, the number of dimensions of the first data arrangement format is P, and P is a positive integer;

[0114] S2, transform the original data arrangement according to the specified transformation scheme to obtain the target data arrangement adapted to the neural network processor. The target data arrangement adopts the second data arrangement format, and the number of dimensions of the second data arrangement format is Q, where Q is a positive integer greater than P. The specified transformation scheme is used to convert the number of dimensions of the data arrangement format from P to Q and eliminate the influence of the transformation process on the calculation results.

[0115] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] Based on the above Figure 3 The method shown and Figure 6 To achieve the above objectives, embodiments of the present invention also provide an electronic device, such as... Figure 7 As shown, it includes a memory 52 and a processor 51, both of which are mounted on a bus 53. The memory 52 stores a computer program, and the processor 51 executes the computer program to implement... Figure 3 The diagram shows a method for optimizing the NPU data layout.

[0117] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a memory (such as a CD-ROM, USB flash drive, portable hard drive, etc.) and includes several instructions to cause an electronic device (such as a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present invention.

[0118] Optionally, the device can also connect to a user interface, network interface, camera, radio frequency (RF) circuitry, sensors, audio circuitry, Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. Optional network interfaces may include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0119] Those skilled in the art will understand that the structure of an electronic device provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0120] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0121] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0122] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0123] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0124] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0125] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0126] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A graph optimization method for NPU data layout, characterized in that, The method includes: Obtain the original data arrangement of operators in a deep learning model, wherein the deep learning model runs on a neural network processor (NPU), and the original data arrangement adopts a first data arrangement format, wherein the number of dimensions of the first data arrangement format is P, and P is a positive integer; The original data arrangement is transformed according to the specified transformation scheme to obtain a target data arrangement adapted to the neural network processor. The target data arrangement adopts a second data arrangement format, and the number of dimensions of the second data arrangement format is Q, where Q is a positive integer greater than P. The specified transformation scheme is used to convert the number of dimensions of the data arrangement format from P to Q and eliminate the influence of the transformation process on the calculation results. The method further includes, while or after transforming the original data arrangement according to a specified transformation scheme to obtain a target data arrangement adapted to the neural network processor (NPU), classifying the operators in the deep learning model into multiple types according to whether the calculation results of the operators in the deep learning model are affected by the transformation; configuring first attribute information for each operator in the deep learning model according to its type, so that the calculation results of the operators are not affected by the transformation.

2. The method according to claim 1, characterized in that, The original data arrangement is transformed according to a specified transformation scheme to obtain a target data arrangement adapted to a neural network processor (NPU), including: Align dimension C in the original data layout to S to obtain the intermediate data layout, where S is 2. n n is a preset positive integer; The intermediate data layout is converted from the first data layout format to the second data layout format to obtain the target data layout.

3. The method according to claim 1, characterized in that, Based on whether the computation results of the operators in the deep learning model are affected by the transformation, the operators in the deep learning model are divided into several types, including: Based on whether the computation results of operators in the deep learning model are affected by transformations, the operators in the deep learning model are divided into three categories: Category I operators, Category II operators, and Category III operators, wherein: The first type of operator is one that does not require operation on the C dimension of the data arrangement and whose result is unaffected; The second type of operator is an operator that needs to operate on the C dimension of the data arrangement and whose result is affected; The third type of operator is an operator used to process a specified data layout format and whose results are affected.

4. The method according to claim 3, characterized in that, Configure first attribute information for each operator of the deep learning model according to its type, so that the computation result of the operator is not affected by the transformation, including: Configure the first attribute information of the i-th operator in the deep learning model, where i is a positive integer less than or equal to the total number of operators K in the deep learning model: If the i-th operator belongs to the first type of operator, set the pad_info of the output tensor of the i-th operator to be the same as the pad_info of the input tensor; pad_info is used by the next operator to determine whether data restoration is required. If the i-th operator belongs to the second type of operator, set the pad_info of the output tensor of the i-th operator to []. If the i-th operator belongs to the third type of operator, set the pad_info of the output tensor of the i-th operator to [C, C2]. n ], [C, C2 n In the figure, C represents the original length of the tensor, C2 n Indicates the length after conversion; If the current value of i is less than the total number of operators K, the value of i is incremented by 1, and the step of configuring the attribute information of the i-th operator in the deep learning model is continued according to the changed value of i. If the current value of i is equal to the total number of operators K, the configuration of the attribute information of the operators in the deep learning model ends.

5. The method according to claim 2, characterized in that, The intermediate data arrangement is converted from the first data arrangement format to the second data arrangement format to obtain the target data arrangement, including: The intermediate data arrangement is parsed according to the first data arrangement format to obtain the data of each dimension in the intermediate data arrangement. The data in each dimension of the intermediate data arrangement is sorted according to the second data arrangement format to obtain the target data arrangement, wherein the data in dimension C of the intermediate data arrangement is equal to the product of the data in each dimension C of the target data arrangement.

6. The method according to any one of claims 2 to 5, characterized in that, Before converting the intermediate data arrangement from the first data arrangement format to the second data arrangement format to obtain the target data arrangement, the method further includes: Configure a second attribute information for each operator of the deep learning model to reduce the addition of transformation nodes to the data arrangement of the input or output.

7. The method according to claim 6, characterized in that, Configure second attribute information for each operator of the deep learning model to reduce the addition of transformation nodes to the data arrangement of input or output, including: Establish a mapping table op_map and a tensor mapping table tensor_map for each operator in the deep learning model. The mapping table of the operator includes the mapping relationship between each position in the input tensor of the operator and each input position of the neural network processor. The tensor mapping table is used to indicate the mapping relationship between the first data layout format and the second data layout format. Based on the characteristics of the operators in the deep learning model, the mapping table, and the tensor mapping table tensor_map, at least one of the following can be selected to be executed: inserting a transform node, modifying the second attribute information, or keeping the second attribute information unchanged.

8. A graph optimization device for NPU data layout, characterized in that, The device includes: An acquisition unit is used to acquire the original data arrangement of operators in a deep learning model, wherein the deep learning model runs on a neural network processor (NPU), and the original data arrangement adopts a first data arrangement format, wherein the number of dimensions of the first data arrangement format is P, and P is a positive integer. An optimization unit is used to transform the original data arrangement according to a specified transformation scheme to obtain a target data arrangement adapted to the neural network processor. The target data arrangement adopts a second data arrangement format, the number of dimensions of the second data arrangement format is Q, and Q is a positive integer greater than P. The specified transformation scheme is used to convert the number of dimensions of the data arrangement format from P to Q and eliminate the influence of the transformation process on the calculation results. The optimization unit is further configured to: while or after converting the original data arrangement according to a specified conversion scheme to obtain a target data arrangement adapted to the neural network processor NPU, classify the operators in the deep learning model into multiple types according to whether the calculation results of the operators in the deep learning model are affected by the conversion; configure first attribute information for each operator in the deep learning model according to its type so that the calculation results of the operators are not affected by the conversion.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images