Method and apparatus for processing operators

Abstract

The embodiments of the present disclosure disclose a method for processing operators, an apparatus for processing operators, an electronic device and a computer-readable storage medium. The method comprises: based on a neural network model, determining a computational graph corresponding to the neural network model; based on the computational graph, determining a search space that enables the neural network model to run, the search space comprising a candidate set of memory layouts of the multiple operators and a candidate set of implementations of the multiple operators; based on the search space, determining an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators; and based on the test results of the intermediate representation on hardware, adjusting the memory layout and implementation of the multiple operators. The embodiments of the present disclosure can improve the performance of the deployed neural network model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically to a method for processing an operator, a device for processing an operator, an electronic device, and a storage medium. Background Art

[0002] Currently, artificial intelligence chips are developing rapidly and have been widely used in various fields, from intelligent driving to medical diagnosis to smart homes. One of the core functions of artificial intelligence chips is to efficiently process neural network models. Neural networks are represented by computational graphs, and the nodes in the computational graph are operators. Operators are the basic units for performing specific computing tasks. The connections between nodes represent tensor data, and the way these data are stored on the hardware, that is, the memory layout, has a direct impact on performance.

[0003] The same operator can be implemented by different memory layouts. Different memory layouts may lead to significant differences in the performance of the same operator. In practical applications, in the same neural network model, the data layout corresponding to the output tensor of the upstream operator and the input tensor of the downstream operator is likely to be inconsistent. When this happens, the tensor data output by the upstream operator needs to be rearranged in memory to adapt to the layout requirements of the downstream operator. However, this process is often accompanied by increased performance overhead because it involves additional data movement and conversion operations.

[0004] Therefore, it is necessary to improve the implementation method of traditional neural network models so as to determine the memory layout of each operator when deploying the neural network model, so as to reduce data reordering and performance loss, thereby improving the operating efficiency of the model. Summary of the invention

[0005] Embodiments of the present disclosure provide a method for processing an operator, an apparatus for processing an operator, an electronic device, and a computer-readable storage medium.

[0006] An embodiment of the present disclosure provides a method for processing operators, comprising: based on a neural network model, determining a computation graph corresponding to the neural network model; based on the computation graph, determining a search space that enables the neural network model to run, the search space comprising a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators; based on the search space, determining an intermediate representation that incorporates memory layout information and implementation method information of the multiple operators; and based on a test result of the intermediate representation on hardware, adjusting the memory layout and implementation methods of the multiple operators.

[0007] For example, determining an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators also includes: determining an upstream operator and a downstream operator among the multiple operators, the output tensor of the upstream operator being the input tensor of the downstream operator; creating a conversion operator for adjusting the memory layout of the output tensor of the upstream operator based on the difference in memory layout between the output tensor of the upstream operator and the input tensor of the downstream operator; and determining an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators based on the conversion operator, the multiple operators including the conversion operator.

[0008] For example, determining the intermediate representation that integrates the memory layout information and implementation information of the multiple operators includes: determining the intermediate representation that integrates the memory layout information and implementation information of the multiple operators based on information of the computational graph corresponding to the neural network model.

[0009] For example, the method further includes: determining executable code on hardware based on the intermediate representation that incorporates memory layout information and implementation information of the multiple operators; and running the executable code in a hardware environment to obtain a test result of the intermediate representation on hardware.

[0010] For example, determining the computational graph corresponding to the neural network model based on the neural network model includes: determining the intermediate representation of the computational graph corresponding to the neural network and the intermediate representation of each operator in the neural network model based on the neural network model, wherein the intermediate representation of the computational graph and the intermediate representation of each operator are integrated into a unified intermediate representation.

[0011] For example, the implementation of the multiple operators includes at least one of the following: automatic code generation, implementation based on an operator library, and implementation based on a programming language.

[0012] For example, the method also includes: determining a training data set based on the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators and a test result of the intermediate representation on hardware; and training a cost model based on the training data set, wherein the cost model is used to determine the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators.

[0013] For example, an embodiment of the present disclosure provides an apparatus for processing operators, comprising: a first module, configured to: determine, based on a neural network model, a computational graph corresponding to the neural network model; a second module, configured to: determine, based on the computational graph, a search space that enables the neural network model to run, the search space comprising a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators; a third module, configured to: determine, based on the search space, an intermediate representation that incorporates memory layout information and implementation method information of the multiple operators; and a fourth module, configured to: adjust the memory layout and implementation methods of the multiple operators based on test results of the intermediate representation on hardware.

[0014] An embodiment of the present disclosure provides an electronic device, including: a processor; and a memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the above-mentioned method for processing an operator is executed.

[0015] An embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned method for processing an operator is implemented.

[0016] According to another aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various aspects or various optional implementations of the above-mentioned various aspects.

[0017] According to the embodiments of the present disclosure, the choice of memory layout of each tensor can be comprehensively considered at the level of the computational graph and the level of operator implementation, thereby improving the performance of the deployed neural network model. In addition, further, according to some embodiments of the present disclosure, it is also possible to flexibly insert conversion operators for implementing changes in the memory layout of tensors between upstream and downstream operators, and optionally further consider the impact of different kernel implementations on the performance of the neural network model. In this way, more efficient data flow and calculation can be achieved, thereby improving the reasoning performance of the entire neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the following briefly introduces the drawings required for describing the embodiment. The drawings described below are only exemplary embodiments of the present disclosure.

[0019] Figure 1A flowchart of a method for processing an operator according to an embodiment of the present disclosure is shown.

[0020] Figure 2 An example of a computation graph according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic diagram of graph-computing fusion according to an embodiment of the present disclosure is shown.

[0022] Figure 4 Another flow chart of a method for processing an operator according to an embodiment of the present disclosure is shown.

[0023] Figure 5 is a schematic diagram showing an apparatus for processing an operator according to an embodiment of the present disclosure.

[0024] Figure 6 is an architectural diagram illustrating a computing device according to an embodiment of the present disclosure.

[0025] Figure 7 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0027] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0028] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0029] Optionally, various models involved in the embodiments of the present disclosure may be artificial intelligence models, especially neural network models based on artificial intelligence. Typically, neural network models based on artificial intelligence are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, a neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a useful representation of the output generated in the output layer. The network nodes (i.e., neurons) are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layer, activation layer, pooling layer, convolution layer, etc. The input and output of the neural network model can take various forms, and the present disclosure is not limited to this.

[0030] Artificial intelligence models are usually composed of computing modules, which are called operators (Operators, Op for short). An operator is a mapping from a function space to a function space O: X→X'; in a broad sense, any operation on a function can be considered an operator. An operator is different from a physical operator. It is a mapping through which an element of a vector space produces another element in another vector space (or possibly the same vector space) through this mapping (or module). Specifically, in the field of deep learning and artificial intelligence, an operator refers to a function that performs a mathematical operation or transformation on a tensor.

[0031] Common operators include: Convolution, Pooling, Fully Connected, Activation Function, Normalization, Loss Function, etc. These operators form the basic computing units of deep neural network models through combination and stacking. Different deep learning frameworks (such as TensorFlow and PyTorch) usually provide optimized operator implementations to improve the efficiency of model training and reasoning.

[0032] Generally speaking, the neural network model to be deployed on the artificial intelligence chip can be represented by a computational graph. A computational graph is a data flow graph in which the nodes represent various mathematical operations, such as matrix multiplication, convolution operations, etc., and the edges represent the direction of data flow. The nodes in the computational graph can be used interchangeably with operators. Operators such as conv2d (two-dimensional convolution), add (addition), relu (rectified linear unit), and concat (connection) are all operators that perform specific computing tasks. The input data flows between these operators in the direction of the edges, and after a series of calculation steps, the output of the model is finally generated. These nodes and the connection relationship between them constitute the graph topology of the model, which describes in detail the data dependency relationship between operators.

[0033] When deploying a neural network model on an AI chip, the compiler (or AI compiler) is responsible for converting the model definition into a computational graph. Traditionally, when processing a computational graph, a memory layout is usually selected at the model input stage, and the memory layout of operators in the entire model is dynamically determined by the input data and data flow. Although in some cases, traditional solutions can achieve memory layout transformations between operators, these memory layout transformations often cannot optimize the performance of AI chips. Traditionally, only the memory layout of a single operator can be selected or adjusted to optimize performance. This solution does not take into account the performance of the entire computational graph or model level, and thus cannot optimize the overall reasoning performance of the neural network model. In addition, multiple operators may be interrelated, and the memory layout of the output tensor of the upstream operator may affect the memory layout of the input tensor of the downstream operator. Improper memory layout selection may lead to degradation of subsequent operator performance.

[0034] In addition, traditional memory layout schemes for model operators often do not fully consider the implementation methods of different operators. In the compiler, there are many ways to implement operators, and the cores generated by different methods are difficult to fuse in many cases, resulting in the choice of memory layout at the model level not taking into account multiple operator implementation methods, thus limiting performance improvement.

[0035] In a traditional approach, it has been proposed to select the memory layout based on the performance of a single operator. This approach does not consider the scenario of transformation operators and other operator fusions used to implement the memory layout change of tensors, and this limitation leads to performance loss.

[0036] In another traditional solution, it is proposed that the memory layout of the input tensor of the neural network model determines the choice of memory layout of each subsequent operator. This solution does not flexibly insert conversion operators for changing the memory layout of tensors on the entire computational graph, so it cannot ensure that the performance of the entire neural network model is optimized. In addition, this solution has not deeply considered the various operator compositions and operator settings within the neural network model.

[0037] It can be seen that the traditional memory layout selection strategy is either selected at the computational graph level or at the operator implementation level, and there are few methods that select the optimal strategy based on both. In order to overcome these limitations, a new method is needed to optimize the deployment of neural network models on hardware.

[0038] Therefore, according to an embodiment of the present disclosure, a method for processing operators is proposed, including: based on a neural network model, determining a computational graph corresponding to the neural network model; based on the computational graph, determining a search space that enables the neural network model to run, the search space including a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators; based on the search space, determining an intermediate representation that incorporates memory layout information and implementation method information of the multiple operators; and based on the test results of the intermediate representation on hardware, adjusting the memory layout and implementation methods of the multiple operators.

[0039] According to the embodiments of the present disclosure, the choice of memory layout of each tensor can be comprehensively considered at the level of the computational graph and the level of the operator, thereby improving the performance of the entire model. In addition, further, according to some embodiments of the present disclosure, it is also possible to flexibly insert conversion operators for implementing changes in the memory layout of tensors between upstream and downstream operators, and optionally further consider the impact of different kernel implementations on the performance of the neural network model. In this way, more efficient data flow and calculation can be achieved, thereby improving the reasoning performance of the entire neural network model.

[0040] Various embodiments of the present disclosure are described in detail through the following drawings.

[0041] Figure 1 A flowchart of a method 10 for processing an operator according to an embodiment of the present disclosure is shown. Figure 2 An example of a computation graph according to an embodiment of the present disclosure is shown. Figure 3 A schematic diagram of graph-computing fusion according to an embodiment of the present disclosure is shown.

[0042] For example, Figure 1 As shown, at least one embodiment of the present disclosure provides a method 10 for processing an operator, which includes the following operations S110 to S140.

[0043] In operation S110, based on the neural network model, a computation graph corresponding to the neural network model is determined.

[0044] like Figure 2 As shown in the figure, the computational graph corresponding to the neural network model can be represented by a graph consisting of a series of nodes. Each node (also called an operator) represents a basic computational operation, such as matrix multiplication or an activation function. The edges between nodes indicate the flow of data and define the dependencies and execution order between operations. The topological structure of the computational graph reflects the organization of these nodes and edges and determines how to perform computational tasks efficiently. The input node serves as the starting point of the calculation and accepts the input data of the model, while the output node produces the final calculation result. Figure 2 In the figure, conv2d (two-dimensional convolution), add (addition), relu (rectified linear unit), and concat (connection) nodes are operators. The input data is calculated by each operator in the direction of the edge, and finally the output of the model is obtained.

[0045] Optionally, a neural network framework (such as PyTorch and TensorFlow) can be used to automatically build a computational graph based on the neural network model. In this automated construction process, the neural network framework can first define the network structure, including the types of layers and the connection methods; then initialize the network parameters. Then, during the training process, the data is forward propagated through the model so that the neural network framework automatically builds a computational graph, in which nodes represent operations and edges represent data flows. Then, back propagation is used to automatically calculate the gradient, and the parameters are updated through the optimization algorithm. Of course, the present disclosure is not limited to this.

[0046] Optionally, operation S110 may include: based on the neural network model, determining an intermediate representation (IR) of a computational graph corresponding to the neural network and an intermediate representation of each operator in the neural network model. The intermediate representation of the computational graph and the intermediate representation of each operator are integrated into a unified intermediate representation. Of course, the present disclosure is not limited to this.

[0047] Among them, the intermediate representation is a hierarchical, abstract, program-like or source code representation of the model. After completing the front-end work (such as building a computational graph), the compiler can first generate its customized intermediate representation, and then execute various analysis and optimization algorithms on this basis, and finally generate the target code. That is, the intermediate representation is an abstract and simplified form of the model in the software components at different levels during the compilation process. The intermediate representation can have different abstract representations at different levels of subgraphs / operators / instructions.

[0048] Optionally, the operation of determining the intermediate representation of the computational graph corresponding to the neural network involves: converting the operators of the model and their topological relationships into functional class function representations, wherein each operator is a node, implements its computational logic through a defined function, input tensors are passed in as parameters, and output tensors are returned as results. These class function representations are also the intermediate representations of the computational layer. Thus, the intermediate representation of the computational layer can describe information such as the functional class function representation corresponding to the coarse-grained operator, the shape of each input tensor, the shape of each output tensor, the data type of each input tensor, the data type of each output tensor, and operator attributes (such as basic information such as the memory layout of the operator).

[0049] Optionally, the operation of determining the intermediate representation of an operator involves: the compiler first registers the computational details and scheduling strategy of each operator, which includes how the operator is executed and how data flows on the hardware. The compiler then expands this information into a finer-grained intermediate representation. For example, the compiler can decompose high-level operator operations into basic programming structures, such as for loops, if control flow statements, and specific tensor operations and corresponding index processing. In this process, basic calculations such as addition, subtraction, multiplication and division of operators are converted by the compiler into executable intermediate representation instructions.

[0050] In order to integrate the operator with the intermediate representation of the computation graph, the embodiment of the present disclosure optionally proposes a unified intermediate representation that can simultaneously represent the coarse-grained network layer structure and the fine-grained operator computation logic. This intermediate representation should be able to simultaneously describe the overall structure of the computation graph and the specific operation of each operator.

[0051] For example, the integrated unified intermediate representation can be located in the same code, and the unified intermediate representation is used to define functions for the operator layer and functions for implementing the computational layer. The compiler can use the unified intermediate representation to define computational blocks, and use axis remapping to implement the flow of tensor data as described in the computational graph, while read and write operations clarify the data dependencies in the computational process. This approach makes the logic of each computational block not only match the memory layout, but also consistent with the data flow path, realizing the unification of computation and data access within the operator layer. In addition, in this code, the input and output interfaces of the neural network model can be defined through the main function, and the shape, data type and memory layout of the input tensor or output tensor of the entire model can be defined. The main function calls specific operator functions through the data flow mechanism, combining the logic of the operator layer with the data flow to form a complete model or computational graph representation.

[0052] In operation S120, based on the computation graph, a search space that enables the neural network model to run is determined, where the search space includes a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators.

[0053] For example, the “search space” refers to the set of all possible configurations and options that need to be explored in order to achieve the optimal operation of a neural network model. In other words, the “search space” covers a variety of memory layout possibilities, including different data storage methods, and other layout schemes optimized for specific hardware. At the same time, the search space also includes various operator implementations, such as hand-written efficient operators, pre-optimized operator library calls, and automatically generated operator code.

[0054] Optionally, the implementation of an operator refers to different schemes for converting operators representing mathematical operations in a computational graph into executable code in an artificial intelligence and machine learning framework. The implementation of operators mainly includes the use of automatic code generation, implementation based on an operator library, and implementation based on a programming language. Automatic code generation uses a compiler to automatically generate optimized execution code based on the computational graph and hardware characteristics, simplifying the development process and improving cross-platform compatibility; the operator library provides a standardized and optimized set of operators to speed up development and ensure certain performance; and the implementation based on a programming language allows developers to go deep into the hardware level and manually write highly optimized operator code. Of course, the present disclosure is not limited to this.

[0055] In operation S130, based on the search space, an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators is determined.

[0056] Optionally, a memory layout mode and an implementation mode of an operator may be arbitrarily or randomly selected from the search space, and an intermediate representation incorporating the memory layout information and implementation mode information of the multiple operators may be determined based on the selected memory layout mode and the implementation mode of the operator. The compiler may directly compile the intermediate representation into a code executable on the hardware and directly run it on the hardware.

[0057] Optionally, the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators is further integrated with the information of the computational graph corresponding to the neural network model. Operation S130 also includes: based on the information of the computational graph corresponding to the neural network model, determining the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators. This determination process is called the graph-computation fusion process.

[0058] Optionally, in order to determine the intermediate representation of the memory layout information and implementation information of multiple operators, the graph-computation fusion process involves analyzing and identifying a set of operators that can be fused from the computational graph of the neural network model, and then exploring different memory layouts and implementations for these operators. Next, all possible combination configurations of layouts and implementations are generated, and these configurations are passed to the compiler, which automatically generates kernels optimized for specific hardware.

[0059] Optionally, the determination of the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators also includes: determining an upstream operator and a downstream operator among the multiple operators, the output tensor of the upstream operator being the input tensor of the downstream operator; creating a conversion operator for adjusting the memory layout of the output tensor of the upstream operator based on the different memory layouts of the output tensor of the upstream operator and the input tensor of the downstream operator; and determining, based on the conversion operator, an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators, the multiple operators including the conversion operator. Of course, the present disclosure is not limited to this.

[0060] Similarly, in the presence of conversion operators, the compiler can also be used to explore different memory layouts and implementations for these conversion operators and other operators in the computation graph. Then, all possible combinations of layouts and implementations are generated and passed to the compiler, which automatically generates kernels optimized for specific hardware.

[0061] Specifically, Figure 3 As shown in the figure, the computational graph of the assumed neural network model includes 5 nodes. Nodes in the computational graph are also called operators. Figure 3 In the example, the first and last nodes are the input and output nodes respectively, and the middle nodes correspond to different operators, such as transpose, reshape, and conv.

[0062] At this stage, the memory layout of the input tensor, the memory layout of the output tensor, and the implementation of the operator of each node can be randomly selected in the search space. Assume that the memory layout of the input tensor of the input node is layout_0. For the transpose node (operator), the memory layout of the input tensor can be fixed to layout_0, and the memory layout of its output tensor can be randomly selected in the search space, which can be selected as layout_1. The implementation of the transpose operator can be automatically generated by the compiler. For the reshape node, the memory layout layout_2 of the input tensor and the memory layout layout_3 of the output tensor may be randomly selected in the search space, and the operator implementation automatically generated by the compiler is also selected. For the conv node, the memory layout layout_4 of the input tensor and the memory layout layout_5 of the output tensor may be randomly selected in the search space, and then the operator library implementation method is used as the implementation method of the operator. The output node selects layout_5 based on the memory layout of the output tensor of the conv node.

[0063] Since the memory layout of the input tensor of the reshape node is different from the memory layout of the output tensor of the transpose node, and the memory layout of the output tensor of the reshape node is different from the memory layout of the input tensor of the conv node, a conversion operator layout_convert can be inserted before and after the reshape node. The memory layout of the input tensor of the conversion operator layout_convert before the reshape node is layout1, and the memory layout of the output tensor is layout2 to match the transpose node and the reshape node. Similarly, the memory layout of the input tensor of the conversion operator layout_convert after the reshape node is layout3, and the memory layout of the output tensor is layout4 to match the reshape node and the conv node.

[0064] After graph-computation fusion, adjacent operators in the computation graph can be merged to optimize memory usage and computational efficiency. For example, the transpose, layout_convert, and reshape nodes can be fused into a fused operator named fused_transpose_reshape_layout_convert, which converts from input layout0 to output layout3, selecting the operator implementation automatically generated by the compiler. The conv node remains independent and continues to select the memory layout layout_4 of the input tensor and the memory layout layout_5 of the output tensor and the implementation based on the operator library. Finally, the output node selects layout_5 based on the memory layout of the output tensor of the cov node.

[0065] After the graph-operation fusion, an intermediate representation that integrates the memory layout information and implementation information of the multiple operators can be obtained. The compiler can directly compile the intermediate representation into code that can be executed on the hardware and run it directly on the hardware.

[0066] In operation S140, based on the test result of the intermediate representation on the hardware, the memory layout and implementation manner of the plurality of operators are adjusted.

[0067] Optionally, the memory layout and implementation of the multiple operators may be adjusted to try to optimize the performance of the neural network model.

[0068] Therefore, according to the embodiments of the present disclosure, the choice of memory layout of each tensor can be comprehensively considered at the level of the computational graph and the level of the operator, thereby improving the performance of the deployed neural network model. In addition, further, according to some embodiments of the present disclosure, it is also possible to flexibly insert conversion operators for implementing changes in the memory layout of tensors between upstream and downstream operators, and optionally further consider the impact of different operator kernel implementations on the performance of the neural network model. In this way, more efficient data flow and calculation can be achieved, thereby improving the reasoning performance of the entire neural network model.

[0069] Figure 4 Another flow chart of the method 10 for processing an operator according to an embodiment of the present disclosure is shown, which shows more details in the method 10 for processing an operator.

[0070] First, in operation S110, based on the neural network model, a computation graph corresponding to the neural network model is determined.

[0071] In operation S110, a front-end intermediate representation conversion operation is optionally performed to convert the neural network model into an internal unified intermediate representation description for subsequent optimization and code generation. The front-end intermediate representation conversion operation is the process of refining the high-level model language into a low-level language, where the intermediate representation of each conversion level has its specific nodes and structure. This process involves traversing each node in the intermediate representation and recursively converting them into a lower-level intermediate representation until the final underlying implementation is reached. At this point, after executing the front-end intermediate representation conversion operation, the intermediate representation of the computational graph corresponding to the neural network and the intermediate representation of each operator in the neural network model use a unified intermediate representation.

[0072] Next, in operation S120, based on the computation graph, a search space that enables the neural network model to run is determined, where the search space includes a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators.

[0073] In the process of determining the search space, the possible memory layout of each operator in the multiple operators in the neural network model and the possible implementation of the operator can be determined first. At this time, the candidate memory layout of the input tensor data and the candidate memory layout of the output tensor that are adapted to each operator can be determined. At the same time, the implementation method that is adapted to each operator can also be determined. Among them, the candidate set of implementation methods corresponding to each operator includes: automatic generation using code, implementation based on operator library, and implementation based on programming language.

[0074] Then, in operation S130, based on the search space, an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators is determined.

[0075] Optionally, in the process of determining an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators, if there is no directly corresponding memory layout conversion implementation between the upstream operator and the downstream operator, a conversion operator can be inserted between the upstream operator and the downstream operator to implement the required memory layout conversion in a combined manner, thereby increasing the possibility of finding an optimal solution.

[0076] Specifically, Figure 4As shown, in operation S130, the operation of automatically inserting a conversion operator can be further performed through the following steps to determine an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators: among the multiple operators, an upstream operator and a downstream operator are determined, and the output tensor of the upstream operator is the input tensor of the downstream operator; based on the different memory layouts of the output tensor of the upstream operator and the input tensor of the downstream operator, a conversion operator is created for adjusting the memory layout of the output tensor of the upstream operator; and based on the conversion operator, an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators is determined, and the multiple operators include the conversion operator.

[0077] Alternatively, if Figure 4 As shown, in operation S130, the graph-calculation fusion operation can be further performed through the following steps to determine an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators: based on the information of the computation graph corresponding to the neural network model, determine an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators. Specifically, the goal of the graph-calculation fusion operation is to maximize the fusion of the intermediate representation of the computation graph, the memory layout information of each operator, and the intermediate representation of the implementation information, reduce the storage and transmission of intermediate data, and improve computational efficiency. As previously mentioned, Figure 3 As described in detail, by identifying operator pairs that can be merged and applying fusion strategies, the number of nodes in the computation graph can be significantly reduced and the data flow path can be simplified.

[0078] Optionally, after operation S130, an executable code on the hardware may be determined based on the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators. Then, the executable code is run in the hardware environment to obtain a test result of the intermediate representation on the hardware. Of course, the present disclosure is not limited to this.

[0079] In operation S140, based on the test result of the intermediate representation on the hardware, the memory layout and implementation manner of the plurality of operators are adjusted.

[0080] The test results on real hardware provide corresponding feedback for adjusting the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators. The test results provide actual runtime data, such as execution time, memory usage, etc., which can be used to evaluate the optimization effect of method 10.

[0081] The test results of the intermediate representation on the hardware provide reference information for further optimizing the intermediate representation. Adjusting the intermediate representation based on the test results can enable the adjusted intermediate representation to be optimized according to the actual operating environment. By benchmarking different hardware platforms, the collected test results reflect the performance of the intermediate representation on specific hardware. For example, some hardware may be better optimized for specific data formats, while other hardware may be more efficient for parallel computing. Based on these test results, the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators can be adjusted in a targeted manner, which may include optimizing the memory layout and implementation of each operator, and adjusting the fusion method between multiple operators to adapt to the parallel processing capabilities of the hardware.

[0082] Optionally, method 10 also includes: determining a training data set based on the intermediate representation that integrates the memory layout information and implementation information of the multiple operators and a test result of the intermediate representation on hardware; and training a cost model based on the training data set, wherein the cost model is used to determine the intermediate representation that integrates the memory layout information and implementation information of the multiple operators.

[0083] Specifically, method 10 can be executed multiple times to repeatedly adjust the intermediate representation, and each adjustment can be verified by hardware testing again to verify its effect, and further iterative optimization can be performed based on the results. The iterative process continuously feeds back to the optimization strategy, and ultimately forms an optimal intermediate representation. The intermediate representation can be used to characterize the neural network model and can achieve optimal performance on the target hardware.

[0084] Combined with neural network technology, a lightweight cost model can be constructed, which is trained on historical data sets derived from the intermediate representations and corresponding test results generated in each iteration. By integrating these historical data into the training of the cost model, the accuracy of the model's prediction of performance overhead can be significantly improved. The core function of the cost model is to learn the complex relationship between the intermediate representations and test results in the historical data set, and predict the performance of different operator configurations based on this.

[0085] The objective function of the cost model is defined as minimizing the consumption of computing resources while meeting the predetermined requirements of model performance. This objective function serves as a reference for the optimization process, guiding the graph-computation fusion operation to make better choices than random choices in the shown space. The cost model comprehensively considers the possibility of operator fusion, the optimization of memory layout, and the choice of operator implementation. By evaluating the performance and resource consumption of different configuration schemes, the cost model can recommend the optimal configuration scheme, thereby achieving efficient operation of the model on the target hardware.

[0086] Therefore, in the embodiments of the present disclosure, the decision of memory layout is no longer limited to the input requirements of a single operator, but considers the combination and configuration between operators from the perspective of the entire model. By automatically inserting conversion operators, the memory layout can be flexibly adjusted to achieve the optimal configuration of the entire model. This choice takes into account the scenario of merging the memory layout conversion operator with other operators to improve overall performance.

[0087] In addition, in the embodiments of the present disclosure, the selection of memory layout involves a variety of implementation methods, including the use of automatic code generation, implementation based on operator libraries, and implementation based on programming languages. These implementation methods provide flexibility and possibility for determining the memory layout. Automatic tuning based on graph-calculation fusion allows searching and tuning the optimal configuration within the entire neural network model, improves the performance of the neural network model, and facilitates the deployment of the neural network model. In the embodiments of the present disclosure, the conversion operator is combined and fused with the original operator in the computational graph, making the memory layout setting and code generation more flexible.

[0088] Figure 5 A schematic diagram of an apparatus 50 for processing an operator according to an embodiment of the present disclosure is shown.

[0089] like Figure 5 As shown, the apparatus 50 for processing an operator according to an embodiment of the present disclosure includes a first module 51 , a second module 52 , a third module 53 , and a fourth module 54 .

[0090] The first module 51 may be configured to: based on the neural network model, determine a computational graph corresponding to the neural network model.

[0091] The second module 52 is configured to: based on the computation graph, determine a search space that enables the neural network model to run, the search space including a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators.

[0092] The third module 53 is configured to: determine an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators based on the search space.

[0093] The fourth module 54 is configured to adjust the memory layout and implementation of the multiple operators based on the test result of the intermediate representation on the hardware.

[0094] In addition, an embodiment of the present disclosure further provides an electronic device, including: a processor; and a memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the above-mentioned method for processing an operator is executed.

[0095] The processor in the embodiment of the present disclosure may be an integrated circuit chip with signal processing capability. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The disclosed methods, operations, and logic block diagrams in the embodiment of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0096] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0097] For example, the method or device according to the embodiment of the present disclosure can also be used by Figure 6 The architecture of the computing device 3000 shown in FIG. Figure 6 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 6 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 6 One or more components of a computing device are shown.

[0098] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Figure 7 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0099] like Figure 7As shown, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] The present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method according to the present disclosure.

[0101] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0103] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for processing an operator, comprising: Based on the neural network model, determining a computational graph corresponding to the neural network model; Based on the computation graph, determining a search space that enables the neural network model to run, the search space comprising a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators; Based on the search space, determine an intermediate representation that incorporates memory layout information and implementation information of the multiple operators; as well as Based on the test results of the intermediate representation on the hardware, adjust the memory layout and implementation of the multiple operators, The determining of the intermediate representation that incorporates the memory layout information and the implementation information of the multiple operators further includes: Among the multiple operators, an upstream operator and a downstream operator are determined, and the output tensor of the upstream operator is the input tensor of the downstream operator; Based on the difference in memory layout between the output tensor of the upstream operator and the input tensor of the downstream operator, creating a conversion operator for adjusting the memory layout of the output tensor of the upstream operator; and Based on the conversion operator, an intermediate representation is determined that incorporates memory layout information and implementation information of the multiple operators, wherein the multiple operators include the conversion operator.

2. The method of claim 1, wherein: The determining of the intermediate representation that incorporates the memory layout information and the implementation information of the multiple operators comprises: Based on the information of the computational graph corresponding to the neural network model, an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators is determined.

3. The method of claim 1, wherein: The method further comprises: Determining executable code on hardware based on the intermediate representation that incorporates the memory layout information and implementation information of the multiple operators; and The executable code is run in a hardware environment to obtain a test result of the intermediate representation on the hardware.

4. The method of claim 1, wherein: The step of determining a computational graph corresponding to the neural network model based on the neural network model includes: Based on the neural network model, an intermediate representation of a computational graph corresponding to the neural network and an intermediate representation of each operator in the neural network model are determined, wherein the intermediate representation of the computational graph and the intermediate representation of each operator are integrated into a unified intermediate representation.

5. The method of claim 1, wherein: The implementation methods of the multiple operators include at least one of the following: automatic code generation, implementation based on an operator library, and implementation based on a programming language.

6. The method of claim 1, further comprising: Determine a training data set based on the intermediate representation fused with the memory layout information and the implementation information of the multiple operators and a test result of the intermediate representation on hardware; as well as Based on the training data set, a cost model is trained, where the cost model is used to determine an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators.

7. A device for processing an operator, comprising: The first module is configured to: determine a computational graph corresponding to the neural network model based on the neural network model; The second module is configured to: determine, based on the computation graph, a search space that enables the neural network model to run, the search space comprising a candidate set of memory layouts of multiple operators and a candidate set of implementation methods of the multiple operators; A third module is configured to: determine, based on the search space, an intermediate representation that incorporates the memory layout information and implementation information of the multiple operators; as well as The fourth module is configured to: adjust the memory layout and implementation of the multiple operators based on the test results of the intermediate representation on the hardware, The determining of the intermediate representation that incorporates the memory layout information and the implementation information of the multiple operators further includes: Among the multiple operators, an upstream operator and a downstream operator are determined, and the output tensor of the upstream operator is the input tensor of the downstream operator; Based on the difference in memory layout between the output tensor of the upstream operator and the input tensor of the downstream operator, creating a conversion operator for adjusting the memory layout of the output tensor of the upstream operator; and Based on the conversion operator, an intermediate representation is determined that incorporates memory layout information and implementation information of the multiple operators, wherein the multiple operators include the conversion operator.

8. An electronic device comprising: processor; as well as A memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the method for processing an operator according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method for processing an operator according to any one of claims 1 to 6.

Images