Neural network global optimization method, device, equipment, medium and product

Abstract

Embodiments of the present application provide a neural network global optimization method, device, equipment, medium and product, relating to the technical field of neural network processing, the method comprising: mining high-frequency subgraphs of a neural network; constructing a neural network model by a neural network architecture search method, adopting a hybrid tensor layout strategy, and outputting data layout; automatically matching the high-frequency subgraphs, adding the set conditions of data flow to the automatic template generation process, and optimizing kernel generation; optimizing the subgraph structures with greater influence by graph mining technology, selecting data layout in advance by developing a performance model, reducing the space of optimization, generating corresponding templates by automatic module generation technology, and optimizing the kernel according to actual parameters, so as to cope with more and more complex calculation graphs, improve the overall effect of graph optimization, and realize efficient global optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural network processing, in particular to a neural network global optimization method, device, equipment, medium and product. BACKGROUND

[0002] At present, automatic optimization technology aiming at mining hardware performance has been widely concerned and researched. The existing tuning work of deep neural network, such as tensor algebra super optimizer TASO, open source machine learning compiler stack TVM, etc., finds that the search space of automatically replacing subgraph structure of the computation graph in the joint tuning process of graph optimization and operator optimization is very huge. When replacing the subgraph, the performance of different equivalent subgraphs needs to be compared, and the evaluation of this performance needs to be based on the actual effect of the kernel. If the automatic optimization kernel is used, kernel tuning needs to be performed on all optimization possibilities, which will bring a huge kernel optimization task, and the overhead is unacceptable. The current scheme is mostly based on the performance of the existing kernel library as the evaluation standard to select the subgraph, and then the overall network with the determined subgraph structure is optimized by the operator. But this optimization scheme divides the computation graph optimization and the operator optimization into two stages and performs them independently, and the computation graph optimization highly depends on the mature operator library on the related platform, which greatly limits the application range of the computation graph automatic optimization. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a neural network global optimization method, device, equipment, medium and product, so as to solve the problem that the strategy space is too large when the computation graph and the operator are jointly optimized in the existing automatic optimization technology aiming at mining hardware performance, which leads to too large tuning overhead, and then it is difficult to find the optimal configuration.

[0004] In a first aspect, the embodiments of the present application provide a neural network global optimization method, comprising:

[0005] The high-frequency subgraph of the neural network is mined by using an open source framework;

[0006] The neural network model is constructed by using a neural network architecture search method, a hybrid tensor layout strategy is used, the data layout is output, and the performance on the CUDA platform is performed;

[0007] The high-frequency subgraph is automatically matched by syntax, the setting conditions of the data flow are added to the automatic template generation process, the kernel is optimized and generated, and stored in the CUDA platform.

[0008] In the implementation process, an open source framework is used to mine high-frequency subgraphs of the neural network; a neural network model is constructed by a neural network architecture search method, a hybrid tensor layout strategy is used, data layout is output and is represented on a CUDA platform; automatic syntax matching is performed on the high-frequency subgraphs, the set conditions of the data flow are added to the automatic template generation process, kernel generation is optimized and is stored on the CUDA platform; by using a graph mining technology, a subgraph structure with a greater impact is optimized, a performance model is developed, data layout is selected in advance, the optimization space is reduced, an automatic module generation technology is used to generate a corresponding template, and kernel optimization is performed according to actual parameters, so as to cope with more and more complex calculation graphs, improve the overall effect of graph optimization, realize efficient global optimization, and mine hardware performance.

[0009] Further, the high-frequency subgraphs of the neural network are mined, including:

[0010] generating subgraphs;

[0011] all subgraph libraries are converted into basic units of high-frequency subgraphs, and the network structure graph is converted by using the same method, and the high-frequency subgraph mining is performed by using a graph mining technology; wherein, an open source framework is used for graph mining work.

[0012] In the implementation process, the open source framework is used for graph mining work, which can quickly find high-frequency subgraphs; in combination with the automatic template generation technology, the high-frequency subgraphs are optimized in a targeted manner, which can meet the deployment requirements of multiple platforms, and can complete the performance evaluation of the subgraphs within a limited automatic optimization time, thereby avoiding the performance loss caused by the fixed library and breaking the boundary between graph optimization and operator optimization.

[0013] Further, the subgraphs are generated, including:

[0014] operators and tensors are taken as inputs, and different operators or tensors are formed into subgraphs through a random process;

[0015] the equivalence of the subgraphs is verified to generate an equivalent replaceable subgraph candidate set;

[0016] wherein, the operator set and the tensor set are defined as the vertices of the graph;

[0017] wherein, the equivalence of the subgraphs is verified, including:

[0018] all generated subgraphs are verified, and if two subgraphs can obtain the same output of the corresponding group under the same data input of different groups, the subgraphs are equivalent.

[0019] In the implementation process, the high-frequency subgraph mining process can quickly and accurately find the main subgraph structure in the optimization process.

[0020] Further, the neural network model is constructed by the neural network architecture search method, a hybrid tensor layout strategy is adopted, and output data layout includes:

[0021] The neural network model is trained based on the automatically optimized kernel data, wherein the input parameters include the number of data layout, input tensor dimension, weight tensor dimension, output tensor dimension and calculation stride;

[0022] According to the set evaluation target, regression fitting is performed;

[0023] The configuration information of each layer in the convolutional network of the neural network model is extracted, and a layout scheme is searched based on a set algorithm, wherein the performance prediction method is used for decision-making during the search process, and the obtained decision result is returned to the convolutional network for tensor layout marking;

[0024] According to the marking, the weight tensor is pre-converted, and the streaming data is pre-operated and layout-converted and output according to the corresponding layer.

[0025] In the above implementation process, the performance difference of the same convolution with different data layouts is evaluated by a suitable method, and a suitable data layout is selected in advance to reduce the optimization space.

[0026] Further, the automatic syntax matching of the high-frequency subgraph includes:

[0027] An abstract syntax analysis tree is constructed by using a programming language Python, and a subgraph JSON file in a subgraph library is analyzed;

[0028] According to the inclusion relationship, the corresponding function is replaced by the syntax of the tensor program generation framework by means of the set mode method of Python, and the optimization of the subgraph is performed through the back end of the tensor program generation framework.

[0029] In the above implementation process, the subgraph is combined, and the subgraph code template meeting the optimality condition is automatically matched and constructed.

[0030] Further, the set condition of the data stream is added to the automatic template generation process, including:

[0031] The optimality condition is added to the set strategy rule of the tensor program generation framework by analyzing the optimization process of the tensor program generation framework;

[0032] When multiple loop unrolling is performed, it is detected whether the parameters meet the optimality condition, and the search range is limited in the given area.

[0033] In the above implementation process, the automatic template generation technology and the set condition are matched to perform efficient optimization of the subgraph structure.

[0034] In a second aspect, an embodiment of the present application provides a neural network global optimization device, comprising:

[0035] a subgraph mining module configured to mine high-frequency subgraphs of the neural network;

[0036] a data layout module configured to construct a neural network model by using a neural network architecture search method, adopt a hybrid tensor layout strategy, and output a data layout;

[0037] a template generation module configured to perform automatic syntax matching on the high-frequency subgraphs, add a set condition of a data flow to an automatic template generation process, and optimize kernel generation.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0039] a processor, a memory, and a bus, wherein the processor is connected to the memory through the bus, and the memory stores computer readable instructions, when the computer readable instructions are executed by the processor, the neural network global optimization method described above is implemented.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed by a server, the neural network global optimization method described above is implemented.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises instructions, when the instructions are executed by a computer, the computer implements the method described above. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 A flowchart of a neural network global optimization method provided by an embodiment of the present application;

[0044] Figure 2 A performance rank comparison diagram of cuDNN and tuning results of a neural network global optimization method provided by an embodiment of the present application;

[0045] Figure 3 A subgraph candidate set construction process diagram of a neural network global optimization method provided by an embodiment of the present application;

[0046] Figure 4 A key operator diagram of subgraph mining of a neural network global optimization method provided by an embodiment of the present application;

[0047] Figure 5 A basic unit process diagram of high-frequency matching of a neural network global optimization method provided by an embodiment of the present application;

[0048] Figure 6 A tensor logical layout and physical layout relationship diagram of a neural network global optimization method provided by an embodiment of the present application;

[0049] Figure 7 A performance performance diagram of AlexNet different convolution layers under different data layouts of a neural network global optimization method provided by an embodiment of the present application;

[0050] Figure 8 A calculation process diagram of a convolution layer of a neural network global optimization method provided by an embodiment of the present application;

[0051] Figure 9 A mixed tensor layout strategy process diagram of a neural network global optimization method provided by an embodiment of the present application;

[0052] Figure 10 A global graph optimization modularization diagram of a neural network global optimization method provided by an embodiment of the present application;

[0053] Figure 11 A structure diagram of a neural network global optimization device provided by an embodiment of the present application;

[0054] Figure 12 A structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0056] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing description, and cannot be understood as indicating or implying relative importance.

[0057] Currently, automatic optimization techniques targeting hardware performance have received extensive attention and research. In terms of theoretical research, although the current research is still relatively scarce, there have been works such as fusion and tiling for loops, which apply polyhedral model transformation mapping to scientifically realize loop changes and propose AKG (Auto Kernel Generator) automatic kernel generation framework; and works such as direct convolution discussing how to design high-reuse hardware based on red and blue pebble model to reduce power consumption problems caused by data movement. These works have been mathematically proven, and theoretically promote the development of deep learning inference optimization technology.

[0058] In terms of implementation methods, more excellent research works have emerged. For example, TASO (Tensor Algebra Super Optimizer) uses equivalent subgraph to automatically search subgraph replacement patterns to explore more hardware-friendly network structures; there are ways to introduce PBQP algorithm for register allocation to replace the conventional dynamic programming algorithm to optimize the combination of data layout of the computation graph; there are ways to introduce reinforcement learning methods to replace general machine learning methods to search for parameters in the parameter optimization space, and so on. At the same time, there are inference engine frameworks such as TVM, TensorComprehensions, XLA, etc., which can provide industry-level automatic optimization for specific hardware.

[0059] In summary, existing researches mainly focus on how to more reasonably utilize the locality and parallelism of data in the theoretical level; and in the method implementation level, more attention is paid to how to convert deep learning inference optimization into an automatic optimization problem and solve this automatic optimization problem through a series of reasonable methods.

[0060] For operator optimization, the current automatic optimization method is mostly based on experience. The general process of automatic optimization of kernels is as follows: first, according to the specific algorithm, a template containing uncertain parameters is designed, and these uncertain parameters determine the performance of the final generated code, that is, the performance of the code will be greatly different with different template parameters. The work of automatic optimization is to find a reasonable parameter combination in these uncertain parameters to generate high-performance code. However, the combination space formed by these parameters is massive. For a simple convolution kernel that uses GPU for calculation, its optimization search space can reach the order of magnitude of hundreds of thousands or even more, and for multiple kernels of a network, the search level is even larger.

[0061] For graph optimization, firstly, the operator fusion mode of graph optimization is less, especially in today's neural network architecture search technology, many fusion modes of the computation graph have not been effectively explored. Secondly, for tensor layout transformation of the computation graph, different shapes of convolution layers exist in the computation graph, and the tensor layout affects the operator efficiency, and different optimal layout features are also presented on different hardware platforms. However, if different tensor layouts are used between different convolution layers, the layout conversion will bring an unacceptable overhead. With the introduction of more and more tensor layout types, how to make a global optimal operator collocation between multiple tensor layouts has always been a difficult problem to solve. The current solution sought by the industry is to combine performance models and global optimization algorithms, use performance models to quickly judge the global performance, and use global optimization algorithms to quickly select the globally optimal layout collocation. However, the effect of this solution has not been effectively broken through in this problem.

[0062] The existing deep neural network optimization work, such as tensor algebra super optimizer TASO, open source machine learning compiler stack TVM, etc., finds that the search space of automatically replacing sub-graph structures of the computation graph is very huge in the joint optimization process of graph optimization and operator optimization. When replacing the sub-graph, the performance of different equivalent sub-graphs needs to be compared, and the performance evaluation needs to be based on the actual effect of the kernel. If an automatic optimized kernel is used, kernel optimization needs to be performed on all optimization possibilities, which will bring a huge kernel optimization task, and the overhead is unacceptable. The current solution is to select the sub-graph based on the performance of the existing kernel library as the evaluation standard, and then optimize the operators of the whole network with the determined sub-graph structure. However, this optimization solution divides the computation graph optimization and the operator optimization into two stages and performs them independently, and the computation graph optimization highly depends on the mature operator library on the related platform, which greatly limits the application range of automatic computation graph optimization.

[0063] In order to solve this problem, it is necessary to introduce automatic optimization of operators into global optimization, but the optimization overhead brought by the superposition of operator kernels and a huge sub-graph optimization space is unacceptable in practical applications, so how to make the joint optimization of the computation graph and the kernel feasible is the key to solving the problem of efficient global optimization of neural networks.

[0064] Based on the above problems, the embodiment of the present application proposes a neural network global optimization method to solve the above problems.

[0065] Please refer to Figure 1 , Figure 1 A flowchart of a neural network global optimization method provided by an embodiment of the present application. The neural network global optimization method comprises:

[0066] 100. Mining high-frequency subgraphs of the neural network.

[0067] It should be noted that, due to the large number of convolution equivalent subgraph candidates, there are more than 700 structures, and each structure may contain 3-4 operators, and each operator has two or three optional data layouts, and more than a dozen. With different tensor shapes, the potential matching space is very large. Understandably, the process of equivalent subgraph matching is an automatic optimization process, which requires a large amount of search, and the search result depends on the comparison of the performance of the corresponding subgraph kernel. If the performance of the kernel depends on the automatically optimized kernel, a large amount of time is required for kernel optimization at each search step, which is not allowed by time, so the current scheme is to use the kernel performance of cuDNN as the evaluation basis for subgraph matching. However, this scheme has two limitations: 1. In different choices, the results of cuDNN cannot be equivalent to the results of automatic optimization, that is, the best scheme determined under the cuDNN standard is not necessarily the best scheme under the automatic optimization kernel standard. As shown in Figure 2 The relative order of the performance of cuDNN in different kernels is different from the relative order of the performance after automatic optimization. 2. cuDNN is only applicable to the NVIDIA GPU platform. One of the functions of automatic optimization technology is to quickly realize the coupling deployment between different algorithms and platforms. cuDNN is a relatively mature deep learning operator library on the NVIDIA platform, but on a large number of intelligent hardware, there is a lack of mature and unified optimization library, which makes the scheme using cuDNN as the evaluation standard applicable range is very small, which seriously limits the landing of automatic optimization technology.

[0068] Figure 2 The relative position in the figure refers to the performance ranking of the kernel corresponding to the 60 groups of parameters according to the cuDNN performance ranking and the tuning optimization performance ranking, and the numbers in the figure represent the performance ranking of the corresponding abscissa (kernel number). That is, the farther apart the two points (circles and triangles) on the same ordinate (such as the ordinate of the kernel number 10), the greater the difference between the relative position ranking of the optimization operator performance and the cuDNN operator performance.

[0069] In view of the two limitations, the embodiment of the present application designs an automatic optimization process of equivalent subgraph matching method, adopts graph mining technology, and the specific process is as follows:

[0070] 110. Generating a subgraph.

[0071] 111. Taking the operator and the tensor as input, and making different operators or tensors form a subgraph through a random process.

[0072] 112. Verify the equivalence of the subgraph to generate an equivalent replaceable subgraph candidate set.

[0073] wherein, the equivalence of the sub-graphs is verified, including: verifying all generated sub-graphs, if two sub-graphs can obtain the same output of the corresponding group under the same data input of different groups, the pair of sub-graphs is equivalent.

[0074] wherein, the operator set and the tensor set are defined as the vertices of the graph.

[0075] Specifically, the high-frequency sub-graphs of the neural network are mined. First, a sub-graph library is generated based on existing work, and the specific generation process is as follows: taking the operators and data tensors as inputs, different operators or tensors are constructed into sub-graphs through a series of random processes, and then the equivalence of the sub-graphs is verified, thereby generating a series of equivalent replaceable sub-graph candidate sets, such as Figure 3 Algorithm 1 in the specification. Wherein, the operator set and the tensor set are defined as the vertices of the graph. Optionally, the candidate sub-graph set of the embodiments of the present application only considers sub-graphs composed of 4 or fewer vertices, and then all generated sub-graphs are verified, and it is considered that two sub-graphs are equivalent if they can obtain the same output of the corresponding group under the same data input of different groups.

[0076] It should be noted that, Figure 4 is the operator (vertex of the graph) and label extracted by the sub-graph candidate set displayed by the embodiments of the present application. These operators are part of the vertices of the sub-graphs, in addition to this, the data tensors are also taken as vertices and defined in the following format: <operator ID><operator ID>, <operator ID><data ID>, which indicates that there is an edge between two operators or an operator and data, and the edge is constructed through Figure 3 Algorithm 1 in the specification. The sub-graph replacement library constructed by Algorithm 1 in the specification will be converted into the above format of the basic input unit for matching, and the high-frequency sub-graph discovery statistics are performed according to the format. The conversion into the basic input unit for matching is shown in Algorithm 2 in the specification, the structure information of the input sub-graph library is outputted, and the basic unit for high-frequency sub-graph mining is outputted, specifically including: initializing an empty model, placing vertices, setting a cursor, and adding edges. Figure 5

[0077] 120, all sub-graph libraries are converted into basic units for high-frequency sub-graphs, and the same method is used to convert the network structure graph, and the high-frequency sub-graphs are mined using graph mining technology; wherein, an open source framework is used for graph mining work.

[0078] Specifically, the embodiments of the present application use an open source framework Peregrine to perform graph mining work, which can quickly discover high-frequency sub-graphs, and combined with automatic template generation technology, the high-frequency sub-graphs are optimized in a targeted manner, so that the deployment requirements of multiple platforms can be met, and the performance evaluation of the sub-graphs can be completed within a limited automatic optimization time, avoiding the performance loss caused by fixed libraries and breaking the boundary between graph optimization and operator optimization.​

[0079] 200. Constructing a neural network model by a neural network architecture search method, adopting a hybrid tensor layout strategy, and outputting data layout.

[0080] After obtaining the high-frequency subgraph, the operator corresponding to each subgraph also has multiple optimization possibilities of data layout. Among them, ReLU (Rectified Linear Unit) and pooling are not sensitive to data layout, and the same data layout as the previous layer can be adopted, while the calculation of the convolution layer is sensitive to data layout. The data of the convolution layer is represented in the form of a tensor in the neural network, and a tensor is a multi-dimensional array structure. Since the memory is a two-dimensional storage structure, for a multi-dimensional array, the tensor will arrange the data in a certain order, and the order of the dimensions will determine the organization form of the data in the memory, that is, correspond to different data layouts. The tensor in the convolution network usually contains four dimensions, the number of feature maps N, the number of feature map channels C, the width W and the height H of the feature map. The permutation combination of these dimensions brings different data layouts, such as NCHW, NHWC, CHWN, etc. As shown in Figure 6 , the logical view shows a four-dimensional tensor logical structure of N = 2 (two independent blocks above and below), C = 2 (two layers of each independent block), W = 4, and H = 3. NCHW and CHWN respectively show the mapping relationship of the tensor logical structure under two data layouts (color texture one-to-one correspondence).

[0081] Since the data layout reflects the arrangement of data in the memory, its difference will affect the data locality feature, and then bring different memory read-write efficiency, thereby affecting the calculation efficiency of the convolution. And this data locality feature, due to the different parameter dimensions of the convolution tensor, is different on different convolution layers, such as Figure 7 , which is the performance of the 5-layer convolution of the deep convolutional neural network AlexNet under two data layouts on the CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model) platform. It can be seen that there is a performance difference among them.

[0082] These performance gaps not only come from the convolution kernel itself, but also come from the tensor layout of the convolution input data and output data, such as Figure 8The relationship of tensor calculation in convolutional neural network is shown. In the convolutional neural network, the tensor is in four-dimensional structure, although the calculation process contains the calculation relationship of 3 tensors with a total of 12 dimensions, but from the figure, it can be found that not all dimensions are independent variables. The output channel Cout depends on the number of convolution kernel Nker, while the channel number Cker of the convolution kernel tensor is generally equal to the channel number Cin of the input tensor, and the output tensor and the input tensor have the following corresponding relationship:

[0083] ;

[0084] ;

[0085] wherein stride represents the span size of two convolution calculations, and represent the height and width of the output respectively, and represent the height and width of the input respectively, and and represent the height and width of the convolution kernel respectively. Since the tensor is a multi-dimensional data structure, and the memory is a two-dimensional data structure, the tensor needs to be converted from high dimension to low dimension to store data in the memory, and different storage orders determine the continuity of data calculation.

[0086] It can be understood that, due to the different data locality performance of different layers, if the optimal data layout of adjacent layers is inconsistent, the performance of certain layers needs to be lost, or layout conversion needs to be performed, but layout conversion will also bring additional time overhead. Therefore, for the entire network, it is necessary to reasonably arrange the data layout of each layer to avoid excessive data layout conversion overhead, so as to fully tap the overall performance of the network. Although there are many research works on the data layout of GPU, how to decide the data layout of network layers from a global perspective is still a challenging problem. Because of the existence of conversion overhead, the global data layout matching is no longer a simple linear superposition problem. In order to reasonably decide the data layout of network layers and characterize the impact of this data layout on the global, the performance of the corresponding layer needs to be predicted, but it is difficult to accurately predict the continuous domain time prediction problem.

[0087] In order to better tap the overall performance of the network and exert the performance of the automatically optimized convolution, it is necessary to determine the most reasonable data layout of each convolution layer, and to make appropriate arrangements for each layer. Therefore, the embodiment of the present application adopts a mixed tensor layout strategy, which allows different convolution layers to select different tensor layouts. However, due to the conversion overhead between different tensor layouts, it is necessary to decide the overall tensor layout from a global level. Since the convolution network is usually deep, the corresponding collocation space is large. In the process of automatic layout planning, it is necessary to judge the performance of the planning scheme as comprehensively and quickly as possible, and then make the optimal decision. The planning algorithm process of the tensor layout is shown in algorithm 3 in Figure 9 .

[0088] It can be found that algorithm 3 can comprehensively consider different situations and terminate the search behavior as soon as possible for inappropriate planning design. In terms of efficiency, when determining the corresponding planning strategy, the performance obtained is the performance prediction problem of each layer. The most natural method to obtain the performance of the tensor layout is to model all memory access operations related to the device and compare them with the quantization time of different tensor layouts. However, such memory access modeling is difficult, and the accuracy is difficult to guarantee because the calculation behavior is also affected by other runtime factors. Therefore, it is difficult to model all operations in actual situations.

[0089] In order to more accurately mark the reasonable data of different layers, the embodiment of the present application adopts a more general method to obtain the determination standard of the tensor layout. At the same time, since it is a global optimization problem, the embodiment of the present application needs to consider the conversion overhead of the tensor layout on many devices, so it is not possible to simply select the best tensor layout of each layer, which requires the standard to be obtained by numerical prediction. It is not a bad method to use a machine learning model for modeling, which has strong generalization ability. However, there are many machine learning methods, and it is difficult for machine learning methods to achieve good performance in numerical prediction. Therefore, the embodiment of the present application selects a neural network architecture search method to construct the model to obtain a high-precision prediction model, and the specific process is as follows:

[0090] 210. Training a neural network model based on the automatically optimized kernel data, wherein the input parameters include the number of data layout, input tensor dimension, weight tensor dimension, output tensor dimension and calculation stride.

[0091] 220. Regression fitting according to the set evaluation target.

[0092] 230. Extracting the configuration information of each layer in the convolution network of the neural network model, and searching for a layout scheme based on the set algorithm, wherein the performance prediction method is used for decision-making in the search process, and the obtained decision result is returned to the convolution network for tensor layout marking.

[0093] 240、According to the mark, the weight tensor is pre-converted, and the streaming data is pre-operated and layout-converted and output according to the corresponding layer.

[0094] The model of the embodiment of the application is trained based on automatically optimized kernel data, and specific input parameters include the number of data layout, input tensor dimension, weight tensor dimension, output tensor dimension, and calculation stride. The R2 score is taken as an evaluation target, and regression fitting is performed. When the convolutional layer parameters are obtained by the embodiment of the application, the performance prediction result under different data layout numbers can be obtained with the help of the model.

[0095] The strategy of the embodiment of the application includes two stages of preprocessing and network optimization. In the preprocessing stage, the configuration information of each layer in the convolutional network is extracted, and layout scheme searching is performed based on algorithm 3. The search process is combined with the performance prediction method to make decisions, and the obtained result is returned to the convolutional network for reasonable tensor layout marking. Then, the device deploying the convolutional network converts the weight tensor in advance according to the marking, and outputs the streaming data after layout conversion in advance according to the corresponding layer.

[0096] 300、Automatically match the syntax of the high-frequency subgraph, add the setting conditions of the data stream to the automatic template generation process, and optimize the kernel generation.

[0097] Optionally, for the template problem, the embodiment of the application introduces an automatic template generation technology, which is proposed by the tensor program generation framework Ansor. In order to better optimize, the embodiment of the application makes two improvements: one is that the embodiment of the application designs automatic syntax matching of subgraph; and the other is that the embodiment of the application adds the optimality conditions of the data stream to the automatic template generation process.

[0098] 310、Use the programming language Python to build an abstract syntax analysis tree, and analyze the subgraph JSON file in the subgraph library.

[0099] 320、According to the inclusion relationship, the corresponding function is replaced with the syntax of the tensor program generation framework by means of the setting mode method of Python, and the optimization of the subgraph is performed through the back end of the tensor program generation framework.

[0100] 330、By analyzing the optimization process of the tensor program generation framework, the optimality conditions are added to the setting strategy rules of the tensor program generation framework.

[0101] 340、When multiple loop unrolling is performed, it is detected whether the parameters meet the optimality conditions, and the search range is limited in the given area.

[0102] Optionally, the embodiment of the present application designs automatic syntax matching of subgraphs. The automatic template technology of Ansor is for the combination of conventional algorithms, while the subgraph applied in the embodiment of the present application is randomly generated. A large number of algorithm combinations that Ansor does not preset are contained therein. Although Ansor can automatically combine templates, the operator order and data input and output of the corresponding subgraph still need to be determined. Therefore, the embodiment of the present application matches different automatic template generation statements according to the characteristics of the subgraph, and constructs an automatic template containing random algorithm combinations for optimization. The embodiment of the present application uses Python to construct a simple abstract syntax analysis tree (AST), analyzes the subgraph JSON file in the subgraph library, and then according to the specific inclusion relationship, with the help of the template (a blueprint or framework for creating documents, code or objects with consistent format or structure) method of Python, the corresponding function is replaced with the syntax of Ansor, and handed over to the Ansor backend for subgraph optimization.

[0103] Optionally, the embodiment of the present application adds the optimality condition of the data flow to the automatic template generation process. The embodiment of the present application analyzes the optimization process of Ansor, adds the optimality condition to the sketch strategy rule of Ansor, detects whether the parameters meet the optimality condition when performing multiple loop unrolling, and thereby limits the search range to a given area.

[0104] The above, the embodiment of the present application mines the high-frequency subgraph of the neural network; constructs a neural network model through a neural network architecture search method, adopts a hybrid tensor layout strategy, and outputs data layout; performs automatic syntax matching on the high-frequency subgraph, adds the setting condition of the data flow to the automatic template generation process, optimizes kernel generation; optimizes the subgraph structure with greater influence through graph mining technology, selects the data layout in advance through the development of the performance model, reduces the optimization space, generates the corresponding template through the automatic module generation technology, and optimizes the kernel according to the actual parameters, so as to cope with more and more complex calculation graphs, improve the overall effect of graph optimization, and realize efficient global optimization.

[0105] For example, the embodiment of the present application proposes a technical optimization route as shown in Figure 10 Three aspects are set: one is to discover possible high-frequency subgraphs through subgraph mining technology, and optimize the subgraph structure with greater influence, as shown in Figure 10 high-frequency subgraph mining. Two is to develop a performance model, evaluate the performance difference of the same convolution with different data layouts through a suitable method, select a suitable data layout in advance, and reduce the optimization space, as shown in Figure 10the convolution performance model in the figure. Third, the subgraph template problem. Since the structural space of subgraph replacement is randomly generated, it may introduce operators that cannot be defined in advance. At this time, the automatic template generation technology is needed to generate the corresponding template, and the kernel optimization is performed according to the actual parameters, so as to cope with more and more complex computation graphs and improve the overall effect of graph optimization, such as Figure 10 the automatic target matching in the figure. Therefore, after reasonable pruning of the optimization space, the efficiency and effect can be taken into account as much as possible, and the upper limit of the overall performance optimization of the computation graph is improved.

[0106] The embodiment of the application focuses on the global optimization problem, breaks the traditional status of independent optimization of operators and computation graphs, reduces the dependence of computation graph optimization on mature operator library, and makes the automatic optimization of computation graph and operator applicable to a wider platform. The embodiment of the application designs a modular solution for global graph optimization, proposes a method based on high-frequency subgraph matching to determine the subgraph structure to be optimized, and cooperates with the automatic template generation technology and optimality condition to perform efficient optimization of the subgraph structure. At the same time, the optimization space of the pruned subgraph is pruned according to the data layout evaluation scheme of the performance model, the efficiency of automatic optimization is improved, the ability of graph and operator fusion optimization is fully released, and good results are achieved. On the CUDA platform, the speedup ratio of the embodiment of the application to the common convolutional neural network inference scene is about 1.3 times.

[0107] The above steps are not executed in the order of the numbers described in strict sequence, and should be understood as a whole scheme.

[0108] Secondly, on the basis of the above embodiment, Figure 11 The structure diagram of a neural network global optimization device provided by the embodiment of the application is shown in FIG. 2. Figure 11 The neural network global optimization device provided by the embodiment of the application specifically includes a subgraph mining module 201, a data layout module 202 and a template generation module 203.

[0109] The subgraph mining module 201 is configured to mine high-frequency subgraphs of the neural network. The data layout module 202 is configured to construct a neural network model by a neural network architecture search method, adopt a hybrid tensor layout strategy, and output a data layout. The template generation module 203 is configured to perform automatic syntax matching on the high-frequency subgraphs, add the set conditions of the data flow into the automatic template generation process, and optimize kernel generation.

[0110] The above, the embodiment of the application mines the high-frequency subgraph of the neural network; a neural network model is constructed through a neural network architecture search method, a mixed tensor layout strategy is adopted, and data layout is output; automatic syntax matching is performed on the high-frequency subgraph, the set conditions of the data flow are added to the automatic template generation process, and kernel generation is optimized; through graph mining technology, the subgraph structure with greater influence is optimized, through the development of a performance model, the data layout is selected in advance, the optimization space is reduced, with the help of automatic module generation technology, the corresponding template is generated, and kernel optimization is performed according to actual parameters, so as to cope with more and more complex calculation graphs, improve the overall effect of graph optimization, and realize efficient global optimization.

[0111] The neural network global optimization apparatus provided by the embodiment of the application can be used to execute the neural network global optimization method provided by the above embodiment, and has corresponding functions and beneficial effects.

[0112] In a third aspect, the embodiment of the application further provides an electronic device, which can integrate the neural network global optimization apparatus provided by the embodiment of the application. Figure 12 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the application. Referring to FIG. 1, Figure 12 The electronic device includes an input apparatus 33, an output apparatus 34, a memory 32, and one or more processors 31; the memory 32 is used to store one or more programs; when the one or more programs are executed by the one or more processors 31, the one or more processors 31 implement the neural network global optimization method provided by the above embodiment. The input apparatus 33, the output apparatus 34, the memory 32, and the processor 31 can be connected through a bus or other means, Figure 12 In the embodiment, the connection through the bus is taken as an example.

[0113] The processor 31 executes various function applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 32, that is, implements the neural network global optimization method described above.

[0114] The electronic device provided above can be used to execute the neural network global optimization method provided by the above embodiment, and has corresponding functions and beneficial effects.

[0115] In a fourth aspect, the embodiment of the application further provides a computer readable storage medium, which includes a stored computer program; when the computer program runs, the computer readable storage medium controls the device where the computer readable storage medium is located to execute the neural network global optimization method described above, and can achieve the same beneficial effects.

[0116] Of course, the computer executable instructions of the storage medium provided by the embodiments of the present application are not limited to the neural network global optimization method described above, and can also perform the related operations in the neural network global optimization method provided by any of the embodiments of the present application.

[0117] In a fifth aspect, the embodiments of the present application further provide a computer program product, and the methods described in the embodiments of the present application can be implemented by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, the computer program product can be implemented in the form of a computer program or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, a core network device, an OAM (Open Application Model) or other programmable devices.

[0118] The computer program or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer program or instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available medium can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; or an optical medium, for example, a digital video disc; or a semiconductor medium, for example, a solid state disk. The computer readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile storage media.

[0119] It should be understood that all the functional modules in the embodiments provided by the present application can be integrated or can exist alone, and two or more functional modules can be integrated to form an independent part.

[0120] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0121] If the functions are realized in the form of software function modules and sold or used as an independent product, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

[0122] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0123] The above descriptions are merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0124] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

Claims

1. A method for global optimization of neural networks, characterized in that, include: We use an open-source framework to mine high-frequency subgraphs in neural networks; A neural network model is constructed using a neural network architecture search method. Combined with the high-frequency subgraphs mined, a hybrid tensor layout strategy is adopted to output the data layout and represent it on the CUDA platform. Automatic syntax matching is performed on high-frequency subgraphs, and the data flow setting conditions are added to the automatic template generation process to optimize kernel generation and store it on the CUDA platform. The method of using an open-source framework to mine high-frequency subgraphs of a neural network includes: Use an open-source framework to generate subgraphs; The generated subgraph includes: Using operators and tensors as inputs, a random process is used to construct subgraphs from different operators or tensors. The equivalence of the subgraphs is verified to generate a set of equivalent replacement subgraph candidates; In this context, the operator set and the tensor set are defined as the vertices of the graph; The hybrid tensor layout strategy includes: The configuration information of each layer in the convolutional network of the neural network model is extracted, and a layout scheme is searched based on a set algorithm. In the process of searching, a performance prediction method is used to make decisions, and the obtained decision results are returned to the convolutional network for tensor layout labeling. The optimal data layout for each convolutional layer is determined, allowing different convolutional layers to choose different tensor layouts, and the overall tensor layout is decided from a global perspective. The data layout reflects how the data is arranged in memory. The performance difference of the data layout comes not only from the convolutional kernel itself, but also from the tensor layout of the convolutional input and output data. The automatic syntax matching of high-frequency subgraphs includes: An abstract syntax tree was constructed using the Python programming language to analyze the JSON files of high-frequency subgraphs in the high-frequency subgraph library. Based on the inclusion relationship, the corresponding functions are replaced with the syntax of the tensor procedural generation framework using Python's setting mode method, and the high-frequency subgraphs are optimized through the backend of the tensor procedural generation framework.

2. The neural network global optimization method of claim 1, wherein, After generating the subgraph, the process also includes: All subgraphs are converted into basic units of high-frequency subgraphs, and the network structure graph is transformed. High-frequency subgraph mining is then performed using graph mining techniques, with an open-source framework employed for the graph mining process.

3. The neural network global optimization method of claim 1, wherein, The verification of the equivalence of the subgraphs includes: Verify all generated subgraphs. If two subgraphs can produce the same output for the corresponding group under the same data input in different groups, then the pair of subgraphs are equivalent.

4. The neural network global optimization method of claim 1, wherein, The method of constructing a neural network model through a neural network architecture search method, employing a hybrid tensor layout strategy, and outputting a data layout includes: The neural network model is trained based on automatically optimized kernel data, where the input parameters include the data layout number, input tensor dimension, weight tensor dimension, output tensor dimension, and computation step. Regression fitting is performed based on the set evaluation objectives; The weight tensors are pre-transformed based on the labels, and the streaming data is output according to the pre-operated layout of the corresponding layer.

5. The neural network global optimization method of claim 1, wherein, Adding the set conditions of the data stream to the automatic template generation process includes: By analyzing the optimization process of the tensor program generation framework, the optimality condition is added to the setting strategy rules of the tensor program generation framework. When performing multiple cycle unrolling, the parameter is detected whether the optimality condition is met, and the search range is limited in the given area.

6. A neural network global optimization apparatus based on the neural network global optimization method according to claim 1, characterized by, Comprise: The subgraph mining module is used to mine high-frequency subgraphs of the neural network by using an open source framework; The data layout module is used to construct a neural network model by a neural network architecture search method, adopt a hybrid tensor layout strategy, output data layout and perform on a CUDA platform; The template generation module is used to automatically match the high-frequency subgraph, increase the setting conditions of the data flow into the automatic template generation process, optimize the kernel generation and store in the CUDA platform; Wherein, the high-frequency subgraph of the neural network is mined by using an open source framework, comprising: Generating subgraph by using open source framework; The generating subgraph comprises: Taking the operator and the tensor as the input, making different operators or tensors constitute a subgraph through a random process; Verifying the equivalence of the subgraph to generate an equivalent replaceable subgraph candidate set; Wherein, the operator set and the tensor set are defined as the vertices of the graph.

7. An electronic device, comprising: Comprise: A processor, a memory and a bus, the processor is connected with the memory through the bus, the memory stores computer readable instructions, when the computer readable instructions are executed by the processor, the neural network global optimization method in any one of claims 1-5 is realized.

8. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the server to realize the neural network global optimization method in any one of claims 1-5.

9. A computer program product, characterised in that, The computer program product comprises instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1-5. The computer program product comprises instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1-5.

Images