Neural network global optimization method and device, equipment, medium and product

Abstract

The embodiment of the invention provides a neural network global optimization method and device, equipment, a medium and a product, and relates to the technical field of neural network processing, and the method comprises the steps: carrying out the mining of a high-frequency sub-graph of a neural network; constructing a neural network model through a neural network architecture search method, and outputting a data layout by adopting a hybrid tensor layout strategy; performing automatic grammar matching on the high-frequency sub-graph, adding a set condition of a data stream to an automatic template generation process, and optimizing kernel generation; the method comprises the following steps: adjusting and optimizing a sub-graph structure with relatively large influence through a graph mining technology, developing a performance model, selecting a data layout in advance, reducing an optimization space, generating a corresponding template by means of an automatic module generation technology, and performing kernel adjustment and optimization according to actual parameters so as to cope with more and more complex calculation graphs and improve the overall effect of graph optimization; and efficient global tuning is realized.

Description

Technical Field

[0001] The present application relates to the field of neural network processing technology, and more specifically, to a neural network global optimization method, device, equipment, medium and product. Background Art

[0002] Existing deep neural network tuning work, such as the tensor algebra super optimizer TASO and the open source machine learning compiler stack TVM, has found that in the process of joint tuning of graph optimization and operator optimization, the search space for automatic subgraph structure replacement of the computational graph is very huge. When performing subgraph replacement, it is necessary to compare the performance of different equivalent subgraphs, and the evaluation of this performance needs to be based on the actual effect of the kernel. If an automatically optimized kernel is used, it is necessary to perform kernel tuning for all optimization possibilities, which will bring a large number of kernel optimization tasks and the overhead is unacceptable. At present, most solutions are based on the performance of existing kernel libraries as evaluation criteria to select subgraphs, and then optimize operators for the overall network with the subgraph structure determined. However, this optimization scheme divides the computational graph optimization and operator optimization into two independent stages, and the computational graph optimization is highly dependent on the mature operator library on the relevant platform, which greatly limits the application scope of automatic optimization of computational graphs. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a neural network global optimization method, device, equipment, medium and product to solve the problem in the prior art that the strategy space is too large when jointly optimizing the computation graph and the operator, resulting in excessive tuning overhead and difficulty in finding the optimal configuration.

[0004] In a first aspect, an embodiment of the present application provides a neural network global optimization method, comprising: Mining high-frequency subgraphs of neural networks; The neural network model is constructed through the neural network architecture search method, and the hybrid tensor layout strategy is adopted to output the data layout; Automatically perform syntax matching on high-frequency subgraphs, add data flow setting conditions to the automatic template generation process, and optimize kernel generation.

[0005] In the above implementation process, the high-frequency subgraphs of the neural network are mined; the neural network model is constructed through the neural network architecture search method, and the hybrid tensor layout strategy is adopted to output the data layout; automatic syntax matching is performed on the high-frequency subgraphs, and the setting conditions of the data flow are added to the automatic template generation process to optimize the kernel generation; the subgraph structure with a greater impact is tuned through graph mining technology, and the data layout is selected in advance by developing a performance model to reduce the optimization space. With the help of automatic module generation technology, the corresponding templates are generated, and the kernel is tuned according to the actual parameters to cope with more and more complex computational graphs, improve the overall effect of graph optimization, and achieve efficient global tuning.

[0006] Furthermore, mining the high-frequency subgraphs of the neural network includes: Generate subgraphs; All sub-graph libraries are converted into basic units of high-frequency sub-graphs, and the same method is used to convert the network structure graph, and graph mining technology is used to mine high-frequency sub-graphs; among them, an open source framework is used for graph mining.

[0007] In the above implementation process, an open source framework is used for graph mining, which can quickly discover high-frequency subgraphs; combined with automatic template generation technology, high-frequency subgraphs are optimized in a targeted manner, which can not only meet the deployment requirements of multiple platforms, but also complete the evaluation of subgraph performance within a limited automatic optimization time, avoiding the performance loss caused by fixed libraries and breaking the boundary between graph optimization and operator optimization.

[0008] Furthermore, the generating a subgraph includes: Taking operators and tensors as input, different operators or tensors form subgraphs through a random process; Verify the equivalence of subgraphs to generate a candidate set of equivalent replaceable subgraphs; Among them, the operator set and tensor set are defined as the vertices of the graph; Among them, the equivalence of subgraphs is verified, including: All generated subgraphs are verified. If two subgraphs can obtain the same output of the corresponding group under the same data input of different groups, the pair of subgraphs are equivalent.

[0009] In the above implementation process, the high-frequency subgraph mining process can quickly and accurately discover the main subgraph structure in the optimization process.

[0010] Furthermore, the neural network model is constructed by a neural network architecture search method, a hybrid tensor layout strategy is adopted, and the output data layout includes: Train a neural network model based on the automatically optimized kernel data, where the input parameters include the data layout number, input tensor dimension, weight tensor dimension, output tensor dimension, and calculation stride; Perform regression fitting according to the set evaluation objectives; Extract the configuration information of each layer in the convolutional network of the neural network model, and search for layout solutions based on the set algorithm. During the search process, the performance prediction method is used to make decisions, and the obtained decision results are returned to the convolutional network for tensor layout marking. The weight tensor is pre-converted according to the tag, and the streaming data is converted and output according to the pre-operation layout of the corresponding layer.

[0011] In the above implementation process, the performance difference of different data layouts for the same convolution is evaluated through appropriate methods, and the appropriate data layout is selected in advance to reduce the optimization space.

[0012] Furthermore, the automatic grammatical matching of high-frequency subgraphs includes: Use the programming language Python to build an abstract syntax analysis tree and analyze the subgraph JSON file in the subgraph library; According to the inclusion relationship, the corresponding function is replaced with the syntax of the tensor program generation framework with the help of Python's setting mode method, and the subgraph is optimized through the backend of the tensor program generation framework.

[0013] In the above implementation process, the subgraph is combined to automatically match and construct a subgraph code template that meets the optimality conditions.

[0014] Furthermore, the step of adding the setting conditions of the data flow to the automatic template generation process includes: By analyzing the optimization process of the tensor program generation framework, the optimality condition is added to the setting strategy rules of the tensor program generation framework; When performing multiple loop unrolling, the parameters are checked to see if they meet the optimality condition, and the search range is limited to a given region.

[0015] In the above implementation process, the automatic template generation technology and setting conditions are combined to efficiently optimize the sub-graph structure.

[0016] In a second aspect, an embodiment of the present application provides a neural network global optimization device, comprising: Subgraph mining module, used to mine high-frequency subgraphs of neural networks; The data layout module is used to build a neural network model through a neural network architecture search method, adopt a hybrid tensor layout strategy, and output data layout; The template generation module is used to automatically match the high-frequency subgraphs with the syntax, add the set conditions of the data flow to the automatic template generation process, and optimize the kernel generation.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the neural network global optimization method as described above.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a server, implements the neural network global optimization method as described above.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions, and when the instructions are executed by a computer, the computer implements the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a process flow of a neural network global optimization method provided in an embodiment of the present application; Figure 2 A schematic diagram showing the performance ranking comparison between cuDNN and the tuning results of a neural network global optimization method provided in an embodiment of the present application; Figure 3 A schematic diagram of a process of constructing a subgraph candidate set for a neural network global optimization method provided in an embodiment of the present application; Figure 4 A schematic diagram of key operators for subgraph mining of a neural network global optimization method provided in an embodiment of the present application; Figure 5 A schematic diagram of a basic unit process for converting high frequency matching of a neural network global optimization method provided in an embodiment of the present application; Figure 6 A schematic diagram of the relationship between the tensor logical layout and the physical layout of a neural network global optimization method provided in an embodiment of the present application; Figure 7A schematic diagram of the performance of different convolutional layers of AlexNet under different data layouts in a neural network global optimization method provided in an embodiment of the present application; Figure 8 A schematic diagram of the calculation process of a convolutional layer of a neural network global optimization method provided in an embodiment of the present application; Fig. 9 A schematic diagram of a hybrid tensor layout strategy process for a neural network global optimization method provided in an embodiment of the present application; Fig.10 A schematic diagram of a global graph optimization modularization of a neural network global optimization method provided in an embodiment of the present application; Fig.11 It is a structural schematic diagram of a neural network global optimization device provided in an embodiment of the present application; Fig.12 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0023] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0024] At present, automatic optimization technology aimed at mining hardware performance has received extensive attention and research. In terms of theoretical research, although the current research is still relatively scarce, there have been some works such as loop fusion and tiling, which apply polyhedral model (polyhedral model) transformation mapping to scientifically implement loop changes and propose the AKG (Auto Kernel Generator) automatic kernel generation framework; for example, for direct convolution, it discusses how to design highly reusable hardware based on the red and blue pebble model to reduce power consumption caused by data movement. These works have been mathematically proven and have theoretically promoted the development of deep learning inference optimization technology.

[0025] At the implementation method level, more excellent research works have emerged. For example, TASO (Tensor Algebra Super Optimizer) uses equivalent subgraphs to automatically search for subgraph replacement patterns to discover more hardware-friendly network structures; there are also practices that introduce the PBQP algorithm for register allocation to replace the conventional dynamic programming algorithm to optimize the data layout of the computational graph; there are practices that introduce reinforcement learning methods to replace general machine learning methods to search parameters in the parameter optimization space, and so on. At the same time, there are inference engine frameworks such as TVM, TensorComprehensions, and XLA that can provide industrial-level automatic optimization for specific hardware.

[0026] In summary, existing research at the theoretical level mainly focuses on how to more reasonably utilize the locality and parallelism of data; while at the method implementation level, it focuses more on how to transform deep learning reasoning optimization into an automatic optimization problem and solve this automatic optimization problem through a series of reasonable methods.

[0027] For operator optimization, the current automatic optimization methods are mostly based on experience. The general process of automatic kernel optimization is to first design a template with uncertain parameters based on the specific algorithm. These uncertain parameters determine the performance of the final generated code. In other words, different template parameters will lead to huge differences in code performance. The work of automatic optimization is to find a reasonable combination of parameters among these uncertain parameters and generate high-performance code. However, the combination space formed by these parameters is massive. For a simple convolution kernel that uses GPU for calculation, its optimization search space can reach tens of millions or even larger. For multiple kernels in a network, the search volume is even larger.

[0028] For graph optimization, first of all, there are fewer operator fusion modes for graph optimization, especially today when neural network architecture search technology is used, many fusion modes of computational graphs have not been effectively explored; then, for the tensor layout transformation of computational graphs, there are convolutional layers of different shapes in the computational graphs, and their tensor layouts affect the operator efficiency, and at the same time, they will present different optimal layout characteristics on different hardware platforms. However, if different tensor layouts are used between different convolutional layers, their layout conversion will bring considerable overhead. With the introduction of more and more tensor layout types, how to make the global optimal operator matching between multiple tensor layouts has always been a difficult problem to solve. The solution currently sought by the industry is to combine performance models with global optimization algorithms, use performance models to quickly judge global performance, and select the global optimal layout matching as soon as possible through global optimization algorithms. However, the effect of such solutions has not been effectively broken through in this regard.

[0029] Existing deep neural network tuning work, such as the tensor algebra super optimizer TASO and the open source machine learning compiler stack TVM, has found that in the process of joint tuning of graph optimization and operator optimization, the search space for automatic subgraph structure replacement of the computational graph is very huge. When performing subgraph replacement, it is necessary to compare the performance of different equivalent subgraphs, and the evaluation of this performance needs to be based on the actual effect of the kernel. If an automatically optimized kernel is used, it is necessary to perform kernel tuning for all optimization possibilities, which will bring a large number of kernel optimization tasks and the overhead is unacceptable. At present, most solutions are based on the performance of existing kernel libraries as evaluation criteria to select subgraphs, and then optimize operators for the overall network with the subgraph structure determined. However, this optimization scheme divides the computational graph optimization and operator optimization into two independent stages, and the computational graph optimization is highly dependent on the mature operator library on the relevant platform, which greatly limits the application scope of automatic optimization of computational graphs.

[0030] In order to solve this problem, it is necessary to introduce the automatic optimization of operators into the global optimization. However, the tuning overhead caused by the superposition of the optimization space of the operator kernel and the sea of ​​quantum graphs is unacceptable in practical applications. Therefore, how to make the joint optimization of the computational graph and the kernel feasible is the key to solving the problem of efficient global tuning of neural networks.

[0031] Based on the above problems, an embodiment of the present application proposes a neural network global optimization method to solve the above problems.

[0032] Please see Figure 1 , Figure 1 A flowchart of a neural network global optimization method provided in an embodiment of the present application. The neural network global optimization method includes: 100. Mining high-frequency subgraphs of neural networks.

[0033] It should be noted that, since there are a large number of candidate convolution equivalent subgraphs, there are more than 700 structures alone, and each structure may contain 3-4 operators. The optional data layout of each operator ranges from two or three to more than a dozen. With different tensor shapes, the potential matching space is huge. It can be understood that the process of equivalent subgraph matching is an automatic tuning process, which requires a lot of searching, and the results of the search depend on the comparison of the performance of the corresponding subgraph kernels. If the performance of the kernel depends on the automatically optimized kernel, each step of the search process will take a lot of time to optimize the kernel, which is not allowed by time, so the current scheme uses the kernel performance of cuDNN as the evaluation basis for subgraph matching. However, this scheme has two limitations: 1. Among different choices, the results of cuDNN cannot be equivalent to the results of automatic optimization, that is, the best solution determined under the cuDNN standard is not necessarily the best solution under the automatic optimization kernel standard. For example Figure 2 As shown in the figure, in different kernels, the relative order of cuDNN performance is different from the relative order of performance after automatic optimization. 2. cuDNN is only applicable to NVIDIA GPU platform. One of the functions of automatic optimization technology is to quickly realize the coupling deployment between different algorithms and platforms. cuDNN is a relatively mature deep learning operator library on the NVIDIA platform, but there is a lack of mature and unified optimization libraries on a large number of intelligent hardware, which makes the scope of application of the solution based on cuDNN as the evaluation standard very small, which seriously limits the implementation of automatic optimization technology.

[0034] Figure 2 The relative ranking in the figure refers to the kernels corresponding to the 60 groups of parameters being sorted according to the cuDNN performance and tuning performance. The numbers in the figure represent the performance ranking of the corresponding horizontal axis (kernel number). That is, the farther the two points (circle and triangle) on the same vertical axis (such as the vertical axis with kernel number 10) are from each other, the greater the difference in relative ranking between the tuning operator performance and the cuDNN operator performance under the parameters.

[0035] In view of these two limitations, the embodiment of the present application designs an automatic optimization process of the equivalent subgraph matching method, using graph mining technology. The specific process is as follows: 110. Generate a subgraph.

[0036] 111. Take operators and tensors as input, and use a random process to make different operators or tensors form a subgraph.

[0037] 112. Verify the equivalence of subgraphs to generate a candidate set of equivalent replaceable subgraphs.

[0038] The equivalence of the subgraphs is verified, including: verifying all generated subgraphs. If two subgraphs can obtain the same output of the corresponding group under the same data input of different groups, the pair of subgraphs are equivalent.

[0039] Among them, the operator set and the tensor set are defined as the vertices of the graph.

[0040] Specifically, the high-frequency subgraphs of the neural network are mined. First, a subgraph library is generated based on existing work. The specific generation process is as follows: operators and data tensors are used as inputs, and different operators or tensors are used to form subgraphs through a series of random processes. Then, the equivalence of the subgraphs is verified, thereby generating a series of equivalent replaceable subgraph candidate sets, such as Figure 3 As shown in Algorithm 1. The operator set and the tensor set are defined as the vertices of the graph. Optionally, the candidate subgraph set of the embodiment of the present application only considers the subgraphs composed of 4 or less vertices, and then all the generated subgraphs are verified. When it is considered that two subgraphs can obtain the same output of the corresponding group under the same data input of different groups, the pair of subgraphs are equivalent.

[0041] Need to explain, Figure 4 It is the operator (graph vertex) and label extracted from the subgraph candidate set shown in the embodiment of this application. These operators are part of the subgraph vertex. In addition, the embodiment of this application also defines the data tensor as a vertex in the following format: <operator ID><operator ID>, <operator ID><data ID>, which means that there is an edge connecting two operators or an operator and data. Figure 3 The subgraph replacement library constructed by Algorithm 1 in will be converted into the subgraph matching basic input unit in the above format, and the discovery statistics of high-frequency subgraphs will be performed according to this format. The way to convert it into the matching basic input unit is as follows Figure 5 As shown in Algorithm 2 in , the structural information of the subgraph library is input, and the basic unit for high-frequency subgraph mining is output, which includes initializing an empty model, placing vertices, setting cursors, and adding edges.

[0042] 120. All sub-graph libraries are converted into basic units of high-frequency sub-graphs, and the same method is used to convert the network structure graph, and graph mining technology is used to mine high-frequency sub-graphs; among them, an open source framework is used for graph mining.

[0043] Specifically, the embodiment of the present application adopts an open source framework Peregrine for graph mining, which can quickly discover high-frequency subgraphs, and combine with automatic template generation technology to optimize high-frequency subgraphs in a targeted manner. Therefore, it can not only meet the deployment requirements of multiple platforms, but also complete the evaluation of subgraph performance within a limited automatic optimization time, avoiding the performance loss caused by fixed libraries and breaking the boundary between graph optimization and operator optimization.

[0044] 200. Construct a neural network model through a neural network architecture search method, adopt a hybrid tensor layout strategy, and output data layout.

[0045] After obtaining the high-frequency sub-graphs, there are also multiple data layout optimization possibilities for the operators corresponding to each sub-graph. Among them, ReLU (Rectified Linear Unit) and pooling are not sensitive to data layout, and the same data layout as the previous layer can be used, while the calculation of the convolution layer is sensitive to data layout. The data of the convolution layer is represented in the form of tensors in the neural network. The tensor is a multi-dimensional array structure. Since the memory is a two-dimensional storage structure, for multi-dimensional arrays, tensors will arrange data in a specific order. The order of arrangement of dimensions will determine the organization of data in memory, which corresponds to different data layouts. The tensors in the convolutional network usually contain four dimensions, the number of feature maps to be processed N, the number of feature map channels C, the feature map width W and the height H. The arrangement and combination of these dimensions result in different data layouts, such as NCHW, NHWC, CHWN, etc. Such as Figure 6 As shown in the figure, the logical view shows a four-dimensional tensor logical structure with N = 2 (two independent blocks above and below), C = 2 (two layers of each independent block stacked), W = 4, and H = 3. NCHW and CHWN respectively show the mapping relationship of the tensor logical structure under two data layouts (one-to-one correspondence between color and texture).

[0046] Since data layout reflects the arrangement of data in memory, its difference will affect the locality of data, which will lead to different memory access and read / write efficiency, thus affecting the computational efficiency of convolution. This data locality will be different in different convolution layers due to the different parameter dimensions of the convolution tensor, such as Figure 7 The figure shows the performance of the deep convolutional neural network AlexNet corresponding to 5 layers of convolution under two data layouts on the CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model) platform. The performance difference can be seen.

[0047] These performance gaps come not only from the convolution kernel itself, but also from the tensor layout of the convolution input data and output data, such as Figure 8 The figure shows the relationship between tensor calculations in convolutional neural networks. In convolutional neural networks, tensors are presented in a four-dimensional structure. Although the calculation process includes the calculation relationship of 3 tensors with a total of 12 dimensions, it can be found from the figure that not all dimensions are independent variables. The output channel Cout depends on the number of convolution kernels Nker, and the number of channels Cker of the convolution kernel tensor is generally equal to the number of channels Cin of the input tensor. The output tensor has the following corresponding relationship with the input tensor: ; ; Among them, stride represents the span size of two convolution calculations. and Represents the height and width of the output, and represent the height and width of the input respectively, and and Represent the height and width of the convolution kernel respectively. Since tensors are multi-dimensional data structures and memory is a two-dimensional data structure, tensors need to be converted from high dimensions to low dimensions to store data in memory, and different storage orders determine the continuity of data calculation.

[0048] It is understandable that due to the different performance of data locality in different layers, if the optimal data layout of adjacent layers is inconsistent, the performance of some layers needs to be lost, or the layout needs to be transformed, but the layout transformation will bring additional time overhead. Therefore, for the entire network, it is necessary to reasonably arrange the data layout of each layer to avoid excessive data layout conversion overhead in order to fully tap the overall performance of the network. Although there are many research works on the data layout of GPUs, how to decide the data layout of the network layer from a global perspective is still a challenging problem. Because of the existence of conversion overhead, the global data layout matching is no longer a simple linear superposition problem. In order to reasonably decide the data layout of the network layer and characterize the impact of this data layout on the global situation, it is necessary to predict the performance of the corresponding layer, but for the time prediction problem in the continuous domain, accurate prediction is more difficult.

[0049] In order to better tap the overall performance of the network and give full play to the performance of the automatically tuned convolution, it is necessary to determine the most reasonable data layout for each layer of convolution, and to adapt to each layer. Therefore, the embodiment of the present application adopts a hybrid tensor layout strategy, allowing different convolution layers to select different tensor layouts. However, due to the conversion overhead between different tensor layouts, it is necessary to decide the matching of the entire tensor layout from a global level. Since convolutional networks are usually deep, their corresponding matching space is larger. In the process of automatic layout planning, it is necessary to judge the performance of the planning scheme as comprehensively and quickly as possible to make the best decision. The planning algorithm process of the tensor layout is as follows: Fig. 9 As shown in Algorithm 3.

[0050] It can be found that by using Algorithm 3, different situations can be considered comprehensively, and the search behavior can be terminated as soon as possible for inappropriate planning designs. In terms of efficiency, it lies in the performance obtained when the corresponding planning strategy is determined, that is, solving the performance prediction problem of each layer. The most natural way to obtain the performance of tensor layout is to model all device-related memory access operations and compare them with the quantization time of different tensor layouts. However, this memory access model is difficult to model, and because the computing behavior is also affected by other factors at runtime, the accuracy is difficult to guarantee. Therefore, it is difficult to model all operations in actual situations.

[0051] In order to more accurately mark the reasonable data of different layers, the embodiment of the present application adopts a more generalized method to obtain the determination standard of the tensor layout. At the same time, since it is a global optimization problem, the embodiment of the present application needs to consider the conversion overhead of the tensor layout on many devices, so it is impossible to simply select the best tensor layout for each layer, which requires that the standard be obtained by numerical prediction. Modeling using a machine learning model is a method with strong generalization ability. However, there are many machine learning methods, and it is difficult for machine learning methods to achieve good performance in numerical prediction. Therefore, the embodiment of the present application chooses to use a neural network architecture search method to construct a model to obtain a high-precision prediction model. The specific process is as follows: 210. Train a neural network model based on the automatically optimized kernel data, wherein the input parameters include a data layout number, an input tensor dimension, a weight tensor dimension, an output tensor dimension, and a computational stride.

[0052] 220. Perform regression fitting according to the set evaluation objectives.

[0053] 230. Extract the configuration information of each layer in the convolutional network of the neural network model, and search for layout solutions based on the set algorithm. In the search process, a performance prediction method is used to make decisions, and the obtained decision results are returned to the convolutional network for tensor layout marking.

[0054] 240. Pre-convert the weight tensor according to the tag, and convert the streaming data according to the pre-operation layout of the corresponding layer and output it.

[0055] The model of the embodiment of the present application is trained based on the kernel data after automatic optimization, and the specific input parameters include the number of data layout, input tensor dimension, weight tensor dimension, output tensor dimension, calculation stride, etc. Regression fitting is performed with R2 score as the evaluation target. When in use, after the embodiment of the present application obtains the relevant convolutional layer parameters, the performance prediction results under different data layout numbers can be obtained with the help of the model.

[0056] The strategy of the embodiment of the present application includes two stages: preprocessing and network optimization. In the preprocessing stage, the embodiment of the present application extracts the configuration information of each layer in the convolutional network, and searches for layout solutions based on Algorithm 3. During the search process, the performance prediction method is used to make decisions, and the obtained results are returned to the convolutional network for reasonable tensor layout marking. After that, the device deploying the convolutional network converts the weight tensor in advance according to the mark, and prepares the layout conversion output of the streaming data in advance according to the corresponding layer.

[0057] 300. Automatically match the high-frequency subgraphs with syntax, add the data flow setting conditions to the automatic template generation process, and optimize the kernel generation.

[0058] Optionally, for the template problem, the embodiment of the present application introduces an automatic template generation technology, which is proposed by the tensor program generation framework Ansor. In order to better optimize, the embodiment of the present application makes two improvements: first, the embodiment of the present application designs automatic syntax matching of subgraphs; second, the embodiment of the present application adds the optimality condition of the data flow to the automatic template generation process.

[0059] 310. Use the programming language Python to build an abstract syntax analysis tree and analyze the subgraph JSON file in the subgraph library.

[0060] 320. According to the inclusion relationship, the corresponding function is replaced with the syntax of the tensor program generation framework with the help of Python's setting mode method, and the subgraph is optimized through the backend of the tensor program generation framework.

[0061] 330. By analyzing the optimization process of the tensor program generation framework, the optimality condition is added to the setting strategy rules of the tensor program generation framework; 340. When performing multiple loop unrolling, check whether the parameters meet the optimality conditions and limit the search range to a given area.

[0062] Optionally, the embodiment of the present application designs automatic syntax matching of subgraphs. Ansor's automatic template technology is combined for conventional algorithms, while the subgraphs used in the embodiment of the present application are randomly generated. This includes a large number of algorithm combinations that Ansor does not have preset. Although Ansor can automatically combine templates, it is still necessary to determine the operator order and data input and output of the corresponding subgraph. Therefore, the embodiment of the present application matches different automatic template generation statements according to the characteristics of the subgraph, and constructs an automatic template containing a random algorithm combination for optimization. The embodiment of the present application uses Python to build a simple abstract syntax analysis tree (AST), analyzes the subgraph JSON file in the subgraph library, and then replaces the corresponding function with Ansor's syntax based on the specific inclusion relationship with the help of Python's template (a blueprint or framework for creating documents, codes, or objects with a consistent format or structure) method, and hands it over to the Ansor backend for subgraph optimization.

[0063] Optionally, the embodiment of the present application adds the optimality condition of the data stream to the automatic template generation process. The embodiment of the present application analyzes the optimization process of Ansor, adds the optimality condition to the sketch strategy rule of Ansor, and detects whether the parameters meet the optimality condition when performing multiple loop expansion, thereby limiting the search range to a given area.

[0064] As described above, the embodiments of the present application mine the high-frequency subgraphs of the neural network; construct a neural network model through a neural network architecture search method, adopt a hybrid tensor layout strategy, and output the data layout; perform automatic syntax matching on the high-frequency subgraphs, add the setting conditions of the data flow to the automatic template generation process, and optimize the kernel generation; use graph mining technology to tune the subgraph structure with a greater impact, develop a performance model, select the data layout in advance, reduce the optimization space, generate the corresponding template with the help of automatic module generation technology, and perform kernel tuning according to the actual parameters to cope with more and more complex computational graphs, improve the overall effect of graph optimization, and achieve efficient global tuning.

[0065] Exemplarily, the present application embodiment proposes the following Fig.10 The technical optimization route shown in the figure has three aspects: First, use subgraph mining technology to discover possible high-frequency subgraphs and optimize the subgraph structures with greater impact, such as Fig.10 The second is to develop a performance model, evaluate the performance differences of different data layouts for the same convolution through appropriate methods, select the appropriate data layout in advance, and reduce the optimization space, such as Fig.10The third is the subgraph template problem. Since the structural space of subgraph replacement is randomly generated, operators that cannot be predefined may be introduced. At this time, it is necessary to use automatic template generation technology to generate corresponding templates and perform kernel tuning according to actual parameters to cope with more and more complex computational graphs and improve the overall effect of graph optimization, such as Fig.10 As shown in the automatic target matching in . Therefore, after reasonable pruning of the tuning space, we can take into account both efficiency and effect as much as possible, and improve the upper limit of the overall performance optimization of the computational graph.

[0066] The embodiments of the present application focus on global optimization problems, breaking the status quo of independent optimization of traditional operators and computational graphs, reducing the dependence of computational graph optimization on mature operator libraries, and making the automatic optimization of computational graphs and operators more applicable to a wider platform. The embodiments of the present application design a set of modular solutions for global graph optimization, propose a method based on high-frequency subgraph matching to determine the subgraph structure to be optimized, and cooperate with automatic template generation technology and optimality conditions to efficiently optimize the subgraph structure. At the same time, according to the data layout evaluation scheme of the performance model, the optimization space of the subgraph is pruned to improve the efficiency of automatic optimization, fully release the ability of graph and operator fusion optimization, and achieve good results. On the CUDA platform, the acceleration ratio of the embodiments of the present application for common convolutional neural network inference scenarios is about 1.3 times.

[0067] The above steps are not to be performed in a strict order as described in the numbers, but should be understood as an overall solution.

[0068] In the second aspect, based on the above embodiments, Fig.11 This is a schematic diagram of the structure of a neural network global optimization device provided in an embodiment of the present application. Fig.11 The neural network global optimization device provided in this embodiment specifically includes: a subgraph mining module 201, a data layout module 202 and a template generation module 203.

[0069] Among them, the subgraph mining module 201 is used to mine the high-frequency subgraphs of the neural network; the data layout module 202 is used to build a neural network model through a neural network architecture search method, adopt a hybrid tensor layout strategy, and output data layout; the template generation module 203 is used to automatically perform syntax matching on high-frequency subgraphs, add the setting conditions of the data flow to the automatic template generation process, and optimize the kernel generation.

[0070] As described above, the embodiments of the present application mine the high-frequency subgraphs of the neural network; construct a neural network model through a neural network architecture search method, adopt a hybrid tensor layout strategy, and output the data layout; perform automatic syntax matching on the high-frequency subgraphs, add the setting conditions of the data flow to the automatic template generation process, and optimize the kernel generation; use graph mining technology to tune the subgraph structure with a greater impact, develop a performance model, select the data layout in advance, reduce the optimization space, generate the corresponding template with the help of automatic module generation technology, and perform kernel tuning according to the actual parameters to cope with more and more complex computational graphs, improve the overall effect of graph optimization, and achieve efficient global tuning.

[0071] The neural network global optimization device provided in the embodiment of the present application can be used to execute the neural network global optimization method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0072] In a third aspect, an embodiment of the present application further provides an electronic device that can integrate the neural network global optimization device provided in an embodiment of the present application. Fig.12 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.12 The electronic device includes: an input device 33, an output device 34, a memory 32 and one or more processors 31; the memory 32 is used to store one or more programs; when the one or more programs are executed by the one or more processors 31, the one or more processors 31 implement the neural network global optimization method provided in the above embodiment. The input device 33, the output device 34, the memory 32 and the processor 31 can be connected by a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0073] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned neural network global optimization method.

[0074] The electronic device provided above can be used to execute the neural network global optimization method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0075] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the neural network global optimization method as described above, and can achieve the same beneficial effects.

[0076] Of course, the storage medium containing computer executable instructions provided in the embodiments of the present application is not limited to the neural network global optimization method described above, and the computer executable instructions can also execute related operations in the neural network global optimization method provided in any embodiment of the present application.

[0077] In a fifth aspect, the embodiments of the present application also provide a computer program product, and the methods described in the various embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the processes or functions described in the various embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model) or other programmable device.

[0078] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0079] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0081] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0082] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0083] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0084] It should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

Claims

1. A neural network global optimization method, characterized in that: include: Mining high-frequency subgraphs of neural networks; The neural network model is constructed through the neural network architecture search method, and the hybrid tensor layout strategy is adopted to output the data layout; Automatically match high-frequency subgraphs with syntax, add data flow settings to the automatic template generation process, and optimize kernel generation; The mining of high-frequency subgraphs of the neural network includes: Generate subgraphs; The generating subgraph comprises: Taking operators and tensors as input, different operators or tensors form subgraphs through a random process; Verify the equivalence of subgraphs to generate a candidate set of equivalent replaceable subgraphs; Among them, the operator set and the tensor set are defined as the vertices of the graph.

2. The neural network global optimization method according to claim 1, characterized in that: After the subgraph is generated, the following step is further included: All sub-graph libraries are converted into basic units of high-frequency sub-graphs, and the network structure graph is converted, and graph mining technology is used to mine high-frequency sub-graphs; among them, an open source framework is used for graph mining.

3. The neural network global optimization method according to claim 1, characterized in that: The verification of the equivalence of the subgraphs includes: All generated subgraphs are verified. If two subgraphs can obtain the same output of the corresponding group under the same data input of different groups, the pair of subgraphs are equivalent.

4. The neural network global optimization method according to claim 1, characterized in that: The neural network model is constructed by the neural network architecture search method, a hybrid tensor layout strategy is adopted, and the output data layout includes: Train a neural network model based on the automatically optimized kernel data, where the input parameters include the data layout number, input tensor dimension, weight tensor dimension, output tensor dimension, and calculation stride; Perform regression fitting according to the set evaluation objectives; Extract the configuration information of each layer in the convolutional network of the neural network model, and search for layout solutions based on the set algorithm. During the search process, the performance prediction method is used to make decisions, and the obtained decision results are returned to the convolutional network for tensor layout marking. The weight tensor is pre-converted according to the tag, and the streaming data is converted and output according to the pre-operation layout of the corresponding layer.

5. The neural network global optimization method according to claim 1, characterized in that: The automatic grammatical matching of high-frequency subgraphs includes: Use the programming language Python to build an abstract syntax analysis tree and analyze the subgraph JSON file in the subgraph library; According to the inclusion relationship, the corresponding function is replaced with the syntax of the tensor program generation framework with the help of Python's setting mode method, and the subgraph is optimized through the backend of the tensor program generation framework.

6. The neural network global optimization method according to claim 5, characterized in that: The step of adding the setting conditions of the data flow to the automatic template generation process includes: By analyzing the optimization process of the tensor program generation framework, the optimality condition is added to the setting strategy rules of the tensor program generation framework; When performing multiple loop unrolling, the parameters are checked to see if they meet the optimality conditions, and the search range is limited to a given region.

7. A neural network global optimization device, characterized in that: include: Subgraph mining module, used to mine high-frequency subgraphs of neural networks; The data layout module is used to build a neural network model through a neural network architecture search method, adopt a hybrid tensor layout strategy, and output data layout; The template generation module is used to automatically match the high-frequency subgraphs with the syntax, add the set conditions of the data flow to the automatic template generation process, and optimize the kernel generation.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the neural network global optimization method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a server, implements the neural network global optimization method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1-6.

Images