Neural network calculation graph optimization method, device and related equipment

By converting the subgraph of the neural network computing graph into a CUDA program, the problem of low optimization efficiency in the existing technology is solved, the optimization process is simplified, and the execution efficiency of the computing graph is improved.

CN116702855BActive Publication Date: 2024-11-19AZURENGINE TECH ZHUHAI INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310474315.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-19
Estimated Expiration
2043-04-27

AI Technical Summary

Technical Problem

The existing neural network computing graph optimization methods have a large workload and low optimization efficiency. They need to predefined a large number of matching templates based on the type and topological relationship of the operator, resulting in an increase in the workload of definition and style matching, which is difficult to improve the execution efficiency of the computing graph.

Method used

Convert the program to be optimized into a target program represented by the output data, establish the corresponding relationship between the output data and the CUDA thread index, and build the CUDA program through the CUDA program conversion module to reduce the dependence on the operator type and topological relationship, and directly Convert to CUDA program.

Benefits of technology

The workload of defining matching templates and style matching is reduced, the optimization efficiency of neural network computing graphs is improved, and the sub-graph optimization process is simplified.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116702855B_ABST
    Figure CN116702855B_ABST
Patent Text Reader

Abstract

The present application provides a method, device and related equipment for optimizing a neural network calculation graph, wherein the method for optimizing the neural network calculation graph includes: converting the program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network calculation graph; establishing a correspondence between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread; according to the target program and the correspondence, converting the program corresponding to the subgraph to be optimized into a CUDA program. The embodiment of the present application does not need to predefine a large number of matching templates according to the type of operator and the topological relationship between operators, and does not need to optimize the subgraph by a large number of style matching with the matching template, but converts the subgraph into a CUDA program, which is conducive to reducing the workload of defining matching templates and style matching, and improving the optimization efficiency of the neural network calculation graph.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and related equipment for optimizing a neural network computation graph. Background Art

[0002] In the field of deep learning, a computation graph is usually used to describe a neural network model. The nodes in the computation graph represent the operators of the neural network, and the lines between the nodes represent the tensors between the operators. In order to improve the execution efficiency of the program code, it is often necessary to optimize the computation graph corresponding to the program code. For this purpose, major manufacturers have launched different computation graph optimization platforms, but their mainstream optimization methods have a relatively large workload and low optimization efficiency. Summary of the invention

[0003] In response to the above problems, the present application provides a method, device and related equipment for optimizing a neural network calculation graph, which is conducive to improving the optimization efficiency of the neural network calculation graph.

[0004] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present application provides a method for optimizing a neural network calculation graph, the method comprising:

[0005] Converting a program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network computation graph;

[0006] Establishing a correspondence between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread;

[0007] According to the target program and the corresponding relationship, the program corresponding to the subgraph to be optimized is converted into a CUDA program.

[0008] In conjunction with the first aspect, in a possible implementation, converting a program corresponding to the subgraph to be optimized into a target program represented by output data of the subgraph to be optimized includes:

[0009] According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained;

[0010] The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

[0011] In combination with the first aspect, in a possible implementation, each node is configured with an index transformation matrix according to the corresponding operator type, the relationship between the index of the input data and the index of the output data of each node is represented by the corresponding index transformation matrix, and the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is represented by an index iteration matrix, and the index iteration matrix is ​​obtained according to the index transformation matrix corresponding to each node.

[0012] In combination with the first aspect, in a possible implementation, according to the target program and the corresponding relationship, converting the program corresponding to the subgraph to be optimized into a CUDA program includes:

[0013] According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed;

[0014] Get the input parameters of the kernel function and get the CUDA program.

[0015] In combination with the first aspect, in a possible implementation, the input parameters of the kernel function include the base address of the input data of the subgraph to be optimized, the base address of the output data, the dimensional information of the input data, the dimensional information of the output data and the elements of the index iteration matrix.

[0016] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0017] Extract multiple consecutive target nodes in the neural network calculation graph; the target node is a node whose index of input data and index of output data satisfy a linear transformation;

[0018] A subgraph consisting of multiple target nodes is determined as a subgraph to be optimized.

[0019] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0020] Compile the CUDA program into optimized nodes through the CUDA compiler;

[0021] The optimized nodes are used to replace the subgraph to be optimized in the neural network calculation graph to obtain the optimized neural network calculation graph.

[0022] A second aspect of an embodiment of the present application provides a device for optimizing a neural network computation graph, the device comprising a CUDA program conversion module; the CUDA program conversion module is configured to:

[0023] Converting a program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network computation graph;

[0024] Establishing a correspondence between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread;

[0025] According to the target program and the corresponding relationship, the program corresponding to the subgraph to be optimized is converted into a CUDA program.

[0026] In conjunction with the second aspect, in a possible implementation, in terms of converting a program corresponding to the subgraph to be optimized into a target program represented by output data of the subgraph to be optimized, the CUDA program conversion module is specifically configured as follows:

[0027] According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained;

[0028] The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

[0029] In combination with the second aspect, in a possible implementation, each node is configured with an index transformation matrix according to the corresponding operator type, the relationship between the index of the input data and the index of the output data of each node is represented by the corresponding index transformation matrix, and the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is represented by an index iteration matrix, which is obtained according to the index transformation matrix corresponding to each node.

[0030] In conjunction with the second aspect, in a possible implementation, in terms of converting the program corresponding to the subgraph to be optimized into a CUDA program according to the target program and the corresponding relationship, the CUDA program conversion module is specifically configured as follows:

[0031] According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed;

[0032] Get the input parameters of the kernel function and get the CUDA program.

[0033] In combination with the second aspect, in a possible implementation, the input parameters of the kernel function include the base address of the input data of the subgraph to be optimized, the base address of the output data, the dimension information of the input data, the dimension information of the output data, and the elements of the index iteration matrix.

[0034] In combination with the second aspect, in a possible implementation manner, the device further includes a sub-image extraction module; the sub-image extraction module is configured to:

[0035] Extract multiple consecutive target nodes in the neural network calculation graph; the target node is a node whose index of input data and index of output data satisfy a linear transformation;

[0036] A subgraph consisting of multiple target nodes is determined as a subgraph to be optimized.

[0037] In combination with the second aspect, in a possible implementation, the device further includes a CUDA program optimization module and a subgraph replacement module;

[0038] The CUDA program optimization module is configured to: compile the CUDA program into optimized nodes through the CUDA compiler;

[0039] The subgraph replacement module is configured to replace the subgraph to be optimized in the neural network calculation graph with the optimized nodes to obtain the optimized neural network calculation graph.

[0040] A third aspect of an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory is used to store one or more programs, and the program includes instructions for executing the method described in the first aspect above.

[0041] A fourth aspect of an embodiment of the present application provides a chip, including: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method described in the first aspect above.

[0042] A fifth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the method described in the first aspect above is implemented.

[0043] A sixth aspect of an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable an electronic device to execute the method described in the first aspect above.

[0044] The above solution of the present application includes at least the following beneficial effects:

[0045] In the embodiment of the present application, the program corresponding to the subgraph to be optimized is converted into a target program represented by the output data of the subgraph to be optimized, and then a corresponding relationship between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread is established, and then the program corresponding to the subgraph to be optimized is converted into a CUDA program according to the target program and the corresponding relationship. Therefore, there is no need to predefine a large number of matching templates according to the type of operators and the topological relationship between operators, and there is no need to optimize the subgraph by matching a large number of styles with the matching templates. Instead, the subgraph is converted into a CUDA program, which is conducive to reducing the workload of defining matching templates and style matching, and improving the optimization efficiency of the neural network calculation graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of the overall process of a related technology provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of subgraph extraction and optimization in a related technology provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a flow chart of a method for optimizing a neural network calculation graph provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of an operator in a neural network calculation graph provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of extracting a subgraph to be optimized provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a grid and thread blocks provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a thread block and threads in a thread block provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of dividing a grid and thread blocks of a kernel function according to dimension information of output data of a subgraph to be optimized provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of compiling a CUDA program provided in an embodiment of the present application;

[0056] Fig.10 A schematic diagram of an optimized neural network calculation graph provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the structure of a neural network calculation graph optimization device provided in an embodiment of the present application;

[0058] Fig.12 A schematic diagram of the structure of another device for optimizing a neural network calculation graph provided in an embodiment of the present application;

[0059] Fig.13A schematic diagram of the structure of another device for optimizing a neural network calculation graph provided in an embodiment of the present application;

[0060] Fig.14 A schematic diagram of the overall process of a neural network calculation graph optimization method provided in an embodiment of the present application;

[0061] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0063] The terms "including" and "having" and any variations thereof that appear in the specification, claims and drawings of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units that are not listed, or may optionally further include other steps or units that are inherent to these processes, methods, products or devices. In addition, the terms "first", "second" and "third" are used to distinguish different objects, rather than to describe a specific order.

[0064] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0065] The following is a description of the related technology of this application in conjunction with the accompanying drawings. The traditional neural network calculation graph optimization method can be as follows Figure 1 As shown, it mainly includes the following steps:

[0066] (1) Find the subgraphs that can be optimized through the subgraph extraction module;

[0067] (2) Optimizing the subgraph through the subgraph optimization module;

[0068] (3) Use the subgraph replacement module to replace the original subgraph with a new one.

[0069] Traditional subgraph extraction and subgraph optimization usually adopt the subgraph pattern matching and mapping method. Figure 2 As shown, according to the operator type and topological relationship of the subgraph, a subgraph style mapping table (including N style mapping templates) is predefined. The subgraph before optimization includes the multiplication (MUL) operation of tensor (Tensor) A and Tensor B, and the addition (ADD) operation of the multiplication result and Tensor C. If the atomic graph finds a matching style in the mapping table, it is replaced with the new subgraph style mapped by it. Exemplarily, the optimized subgraph is replaced with the multiply accumulate (MAC) operation of Tensor A, Tensor B and Tensor C. However, the traditional method has the following disadvantages:

[0070] (1) It is necessary to predefine matching templates based on the operator types and the topological relationships between operators. As the number of operator types increases, it is difficult to exhaust all the style combinations between operators, and the workload of template definition increases.

[0071] (2) The old and new sub-graph styles in the sub-graph style mapping table need to be manually defined by the graph compiler developer, which is a lot of work.

[0072] The related art has not proposed a suitable method to overcome the above shortcomings, so that the problem of low optimization efficiency of neural network calculation graphs has not been well solved.

[0073] In order to overcome the defects or deficiencies in the related art, the present application embodiment provides a method for optimizing a neural network calculation graph. Figure 3 , Figure 3 A schematic diagram of a process flow of a neural network calculation graph optimization method provided in an embodiment of the present application, which method can be applied to electronic devices, such as Figure 3 As shown, the method may include the following steps:

[0074] 301: Convert the program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network calculation graph.

[0075] In the embodiment of the present application, before step 301, the method further includes extracting a subgraph to be optimized from the neural network calculation graph. Specifically, extracting a plurality of continuous target nodes in the neural network calculation graph, wherein the target node is a node whose index of input data and index of output data satisfy a linear transformation, and determining a subgraph consisting of the plurality of target nodes as a subgraph to be optimized.

[0076] For example, the output data and the input data are Tensors (that is, the input and output data in the embodiment of the present application include Tensors). Figure 4 The operator shown, T i is the input Tensor of the operator, T o is the output Tensor of the operator, and F() is the calculation function of the operator. Then the operator can be expressed as:

[0077] T o =F(T i )

[0078] Define the index of the output Tensor element as (x d ,y d , z d ), the value of this element is T o (x d ,y d , z d );

[0079] Define the index of the input Tensor element as (x s ,y s , z s ), the value of this element is T i (x s ,y s , z s );

[0080] Based on the representation of this operator, we have formula (1):

[0081] T o (x d ,y d , z d )=F(T i (x s ,y s , z s )) (1)

[0082] If (x d ,y d , z d ) and (x s ,y s , z s ) satisfies formula (2), then the output data and input data of the operator satisfy the index linear transformation, that is, the index of the output data and the index of the input data satisfy the linear transformation:

[0083]

[0084] Among them, the matrix is the index transformation matrix of the node corresponding to the operator, and the matrix elements are all constants. The matrix element values ​​can be determined by configuration information during compilation. The node whose index satisfies the linear transformation can be determined as the target node. The index transformation matrix of each target node can be pre-configured, for example, Figure 5 In the neural network calculation diagram shown:

[0085] The operator corresponding to the first node is Resize, and its scaling factor is defined as (f x , f y , 1), then the index transformation matrix of the node can be:

[0086]

[0087] The operator corresponding to the second node is transpose, so the index transformation matrix of the node can be:

[0088]

[0089] The operator corresponding to the third node is Scale, so the index transformation matrix of this node can be:

[0090]

[0091] in, Figure 5 T1, T2, T3, T4, and T5 represent five tensors respectively. The first three nodes all satisfy formula (2). Therefore, the first three nodes can be extracted as target nodes, and the subgraph formed by the first three nodes is determined as the subgraph to be optimized. The operator corresponding to the fourth node is convolution. Since one element of the output Tensor of this node needs to be convolved with multiple elements of the input Tensor, this node does not satisfy the index linear transformation. It should be understood that Figure 5 Taking multiple consecutive target nodes as an example, the subgraph extraction method provided in the embodiment of the present application is also applicable to the extraction of a single target node. In this implementation, the extraction of the subgraph to be optimized is based on the indexing of the input data and the output data, and the type of operator is no longer considered.

[0092] Exemplarily, converting a program corresponding to the subgraph to be optimized into a target program represented by output data of the subgraph to be optimized includes:

[0093] According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained;

[0094] The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

[0095] It should be understood that since each node in the subgraph to be optimized is configured with an index transformation matrix according to the corresponding operator type, as shown in formula (2), the relationship between the index of the input data and the index of the output data of each node can be represented by the corresponding index transformation matrix. Define the subgraph to be optimized to include n nodes, H n is the index transformation matrix of the nth node. Define (x o ,y o , z o ) is the index of the output Tensor element of the subgraph to be optimized, (x i ,y i , z i ) is the index of the input TensorElement of the subgraph to be optimized, then according to formula (2), the index of the input data and the index of the output data of n nodes are iterated, and we have:

[0096]

[0097] Let the index iterate over the matrix:

[0098] H c =H 1 H 2 ...H n (4)

[0099] but Figure 5 The index iteration matrix H of the subgraph to be optimized is shown c Can be:

[0100]

[0101] Then formula (3) can be expressed as:

[0102]

[0103] That is, iterate the matrix H by indexing c Represents the relationship between the index of the input data and the index of the output data of the subgraph to be optimized.

[0104] Define the output Tensor element of the subgraph to be optimized as T o (x o ,y o , z o );

[0105] Define the input Tensor element of the subgraph to be optimized as T i (xi ,y i , z i );

[0106] F n () is the calculation function of the nth node.

[0107] Then, according to formula (1) and formula (5), the calculation of the subgraph to be optimized can be converted to:

[0108] var 1 =F 1 (T i [H c (x o ,y o , z o )]) (6)

[0109] var 2 =F 2 (var 1 ) (7)

[0110] var 3 =F 3 (var 2 ) (8)

[0111] T o (x o ,y o , z o )=F N (var n-1 ) (9)

[0112] Among them, var 1 Represents the output Tensor of the first node in the subgraph to be optimized, var 2 Represents the output Tensor of the second node in the subgraph to be optimized, var 3 Represents the output Tensor of the third node in the subgraph to be optimized, var n-1 Represents the output Tensor of the n-1th node in the subgraph to be optimized.

[0113] It can be seen from formula (6) to formula (9) that in the converted target program: the input and output of the subgraph to be optimized are no longer Tensors, but Tensor elements, that is, the program in units of Tensor is converted into a program in units of TensorElement; the intermediate nodes of the subgraph to be optimized become temporary variables, and the input and output of the intermediate nodes can be stored in registers without writing to memory; the Tensor elements in the subgraph to be optimized can all use the index (x) of the output Tensor element of the subgraph to be optimized o ,y o , z o )express.

[0114] 302: Establish a correspondence between the index of the output data of the subgraph to be optimized and the compute unified device architecture CUDA thread index.

[0115] In the embodiments of the present application, it should be understood that a neural network calculation graph program describes operations in units of Tensor, while a Compute Unified Device Architecture (CUDA) program describes operations in units of threads. Therefore, the key to converting a neural network calculation graph to a CUDA program is to find the corresponding relationship between each Tensor element and thread.

[0116] In CUDA, a kernel function (kemel) corresponds to a grid (grid). A grid is composed of multiple thread blocks (threadblocks), and a thread block is composed of multiple threads. The relationship between grid and thread block can be as follows: Figure 6 As shown, (G x , G y , G z ) represents the grid’s dimensional information in the x, y, and z directions. (D x , D y , D z ) represents the dimensional information of threadblock in the x, y, and z directions. The relationship between thread block and thread can be as follows: Figure 7 As shown, Figure 7 The black square in the thread block represents a thread in the thread block. x , t y , t z ) represents the index of the threads in the thread block in the x, y, and z directions.

[0117] Define (X, Y, Z) as the dimension information of the output Tensor of the subgraph to be optimized;

[0118] Definition (d x , d y , d z ) is the index of the thread block in the kernel in the grid;

[0119] If Figure 8 As shown, the grid and threadblock of Kemel are divided according to the dimension information of the output Tensor of the subgraph to be optimized, as follows:

[0120] x o =t x +d x D x (10)

[0121] y o =t y +d y D y (11)

[0122] z o =t z +d z D z (12)

[0123] X=G x D x (13)

[0124] Y=G y D y (14)

[0125] Z=G z D z (15)

[0126] That is, the correspondence between Tensor element and thread can be shown as formula (10)-formula (12).

[0127] 303: According to the target program and the corresponding relationship, convert the program corresponding to the subgraph to be optimized into a CUDA program.

[0128] Exemplarily, according to the target program and the corresponding relationship, converting the program corresponding to the subgraph to be optimized into a CUDA program includes:

[0129] According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed;

[0130] Get the input parameters of the kernel function and get the CUDA program.

[0131] Substituting formula (10)-formula (12) into formula (6)-formula (9) can construct a kernel in thread units. For a kernel, if it is called, its input parameters (kernel parameters) need to be determined. In the embodiment of the present application, the base address Ti of the input data of the subgraph to be optimized, the base address To of the output data, and the dimension information (D xi , D yi , D zi ), output data dimension information (D xo , D yo , D zo ) and the index iteration matrix H c The elements of are used as input parameters of the constructed kernel, and the complete kernel can be as follows:

[0132]

[0133]

[0134] Where blockIdx represents the index of threadblock in the kernel, blockDim represents the dimension information of threadblock in the kernel, and threadIdx represents the index of thread in threadblock. i +y i *D xi , +z i *D yi D xi " indicates that according to the index of the input data (x i ,y i , z i ) and the dimensionality information of the input data (D xi , D yi , D zi ) calculates the offset of the input data, "Ti[offset]" means reading the input data from the memory according to the base address Ti and offset of the input data. "val1*scale+bias" means scaling the input data (for example: Figure 5 The output node of the subgraph to be optimized), "x o +y o *D xo , +z o *D yo D xo " indicates that according to the index of the output data (x o ,y o , z o) and the dimension information of the output data (D xo , D yo , D zo ) calculates the offset of the output data, and "To[offset]" means writing the output data into memory according to the base address To and offset of the output data. x 、f y Represents the index iteration matrix H c elements.

[0135] For the converted CUDA program, that is, the built kernel, the dimension information of the grid and thread block can be set according to formulas (13)-(15). For example, if the dimension information of the thread block in the x direction is 32, then:

[0136] blocksPerGrid.x = D xo / threadsPerBlock.x;

[0137] Among them, blocksPerGrid.x represents the dimension information of the grid in the x direction, threadsPerBlock.x represents the dimension information of the thread block in the x direction, and the dimension information in the y and z directions can be deduced accordingly.

[0138] Exemplarily, the method further includes:

[0139] Compile the CUDA program into optimized nodes through the CUDA compiler;

[0140] The optimized nodes are used to replace the subgraph to be optimized in the neural network calculation graph to obtain the optimized neural network calculation graph.

[0141] In the present application embodiment, Fig. 9 As shown in FIG. 1 , the converted CUDA program is compiled by the CUDA compiler to obtain a callable node (i.e., an optimized node). Figure 5 By replacing the subgraph to be optimized in Fig.10 The optimized neural network calculation graph shown in FIG. CUDA compiler can be an existing compiler.

[0142] In the converted CUDA program, the input and output data of the subgraph to be optimized are stored in memory, and the input and output of the intermediate nodes are stored in registers. This can reduce the amount of computation and memory access during program calling and execution, thereby improving the operating efficiency of the optimized neural network computation graph.

[0143] It can be seen that in the embodiment of the present application, by converting the program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized, and then establishing a correspondence between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread, the program corresponding to the subgraph to be optimized is converted into a CUDA program according to the target program and the correspondence. Therefore, there is no need to predefine a large number of matching templates according to the type of operators and the topological relationship between operators, and there is no need to optimize the subgraph by matching a large number of styles with the matching templates. Instead, the subgraph is converted into a CUDA program, which is conducive to reducing the workload of defining matching templates and style matching, and improving the optimization efficiency of the neural network calculation graph.

[0144] The method of the embodiment of the present application is described above, and the device of the embodiment of the present application is provided below.

[0145] See also Fig.11 , Fig.11 A schematic diagram of the structure of a neural network calculation graph optimization device 1100 provided in an embodiment of the present application. The neural network calculation graph optimization device 1100 is applied to an electronic device or a chip in an electronic device, and the neural network calculation graph optimization device 1100 includes a CUDA program conversion module 1101; the CUDA program conversion module 1101 is configured as follows:

[0146] Converting a program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network computation graph;

[0147] Establishing a correspondence between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread;

[0148] According to the target program and the corresponding relationship, the program corresponding to the subgraph to be optimized is converted into a CUDA program.

[0149] In a possible implementation, in terms of converting a program corresponding to the subgraph to be optimized into a target program represented by output data of the subgraph to be optimized, the CUDA program conversion module 1101 is specifically configured as follows:

[0150] According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained;

[0151] The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

[0152] In one possible implementation, each node is configured with an index transformation matrix according to the corresponding operator type, the relationship between the index of the input data and the index of the output data of each node is represented by the corresponding index transformation matrix, and the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is represented by an index iteration matrix, which is obtained according to the index transformation matrix corresponding to each node.

[0153] In a possible implementation, in terms of converting the program corresponding to the subgraph to be optimized into a CUDA program according to the target program and the corresponding relationship, the CUDA program conversion module 1101 is specifically configured as follows:

[0154] According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed;

[0155] Get the input parameters of the kernel function and get the CUDA program.

[0156] In a possible implementation, Fig.12 As shown, the neural network calculation graph optimization device 1100 also includes a subgraph extraction module 1102; the subgraph extraction module 1102 is configured as follows:

[0157] Extract multiple consecutive target nodes in the neural network calculation graph; the target node is a node whose index of input data and index of output data satisfy a linear transformation;

[0158] A subgraph consisting of multiple target nodes is determined as a subgraph to be optimized.

[0159] In a possible implementation, Fig.13 As shown, the neural network calculation graph optimization device 1100 also includes a CUDA program optimization module 1103 and a subgraph replacement module 1104;

[0160] The CUDA program optimization module 1103 is configured to: compile the CUDA program into optimized nodes through a CUDA compiler;

[0161] The subgraph replacement module 1104 is configured to replace the subgraph to be optimized in the neural network calculation graph with the optimized nodes to obtain the optimized neural network calculation graph.

[0162] based on Fig.13 The overall flow diagram of the neural network calculation graph optimization method provided by the embodiment of the present application can be shown as follows: Fig.14 shown.

[0163] It should be noted that the implementation of each step in the method embodiment can refer to Figure 3-Figure 10The implementation of each step in the method embodiment shown, of course, the optimization device 1100 for the neural network calculation graph provided in the embodiment of the present application includes but is not limited to the above-mentioned unit modules, for example: the optimization device 1100 for the neural network calculation graph may also include a storage unit, and the storage unit may be used to store the program code and data of the optimization device 1100 for the neural network calculation graph.

[0164] It can be seen that in Figure 11-13 In the optimization device 1100 of the neural network calculation graph shown, the program corresponding to the subgraph to be optimized is converted into a target program represented by the output data of the subgraph to be optimized, and then a corresponding relationship between the index of the output data of the subgraph to be optimized and the unified computing device architecture CUDA thread index is established, and then the program corresponding to the subgraph to be optimized is converted into a CUDA program according to the target program and the corresponding relationship. Therefore, there is no need to predefine a large number of matching templates according to the type of operators and the topological relationship between operators, and there is no need to optimize the subgraph by matching a large number of styles with the matching templates, but the subgraph is converted into a CUDA program, which is conducive to reducing the workload of defining matching templates and style matching, and improving the optimization efficiency of the neural network calculation graph.

[0165] See also Fig.15 , Fig.15 The structural diagram of an electronic device 1500 provided in an embodiment of the present application is shown, and the electronic device 1500 includes a processor 1501, a memory 1502, and a communication interface 1503, and the processor 1501, the memory 1502, and the communication interface 1503 are interconnected via a bus 1504. Exemplarily, the electronic device 1500 may be a terminal device or a chip in a terminal device.

[0166] The memory 1502 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 1502 is used for related computer programs and data. The communication interface 1503 is used to receive and send data.

[0167] The processor 1501 may be one or more central processing units (CPUs). When the processor 1501 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0168] The processor 1501 in the electronic device 1500 is used to read the computer program code stored in the memory 1502 and perform the following operations:

[0169] Converting a program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network computation graph;

[0170] Establishing a correspondence between the index of the output data of the subgraph to be optimized and the index of the unified computing device architecture CUDA thread;

[0171] According to the target program and the corresponding relationship, the program corresponding to the subgraph to be optimized is converted into a CUDA program.

[0172] In a possible embodiment, the processor 1501 executes the conversion of the program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized, including:

[0173] According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained;

[0174] The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

[0175] In a possible embodiment, each node is configured with an index transformation matrix according to the corresponding operator type, the relationship between the index of the input data and the index of the output data of each node is represented by the corresponding index transformation matrix, and the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is represented by an index iteration matrix, which is obtained according to the index transformation matrix corresponding to each node.

[0176] In a possible embodiment, the processor 1501 converts the program corresponding to the subgraph to be optimized into a CUDA program according to the target program and the corresponding relationship, including:

[0177] According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed;

[0178] Get the input parameters of the kernel function and get the CUDA program.

[0179] In a possible embodiment, the input parameters of the kernel function include the base address of the input data of the subgraph to be optimized, the base address of the output data, the dimension information of the input data, the dimension information of the output data and the elements of the index iteration matrix. In a possible embodiment, the processor 1501 is further configured to execute:

[0180] Extract multiple consecutive target nodes in the neural network calculation graph; the target node is a node whose index of input data and index of output data satisfy a linear transformation;

[0181] A subgraph consisting of multiple target nodes is determined as a subgraph to be optimized.

[0182] In a possible embodiment, the processor 1501 is further configured to execute:

[0183] Compile the CUDA program into optimized nodes through the CUDA compiler;

[0184] The optimized nodes are used to replace the subgraph to be optimized in the neural network calculation graph to obtain the optimized neural network calculation graph.

[0185] It should be noted that the implementation of each operation can also refer to Figure 3-Figure 10 The corresponding description of the method embodiment shown.

[0186] The present application also provides a chip, including: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned Figure 3-Figure 10 The method according to any one of the embodiments. The chip may be a chip in an electronic device.

[0187] The present application also provides a computer-readable storage medium (Memory), which stores a computer program. When the computer program is executed, the above-mentioned Figure 3-Figure 10 The method described in any one of the embodiments. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the device and the extended storage medium supported by the device. The computer-readable storage medium provides a storage space, which stores the operating system of the device. In addition, one or more computer programs suitable for being loaded and executed by the processor of the device are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0188] The present application also provides a computer program product, the computer program product comprising: a computer program code, when the computer program code is executed, Figure 3-Figure 10 The method process described in any one of the embodiments is implemented.

[0189] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] It should be understood that the processor mentioned in the embodiments of the present application may be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0191] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a ROM, a programmable read-only memory (Programmable ROM, PROM), an EPROM, an electrically erasable programmable read-only memory (ElectricallyEPROM, EEPROM) or a flash memory. The volatile memory may be a RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM).

[0192] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor.

[0193] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0194] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0195] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely exemplary. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0198] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0199] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0200] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0201] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing a neural network computation graph, characterized in that: The method comprises: Converting a program corresponding to a subgraph to be optimized into a target program represented by output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network computation graph; Establishing a correspondence between the index of the output data of the subgraph to be optimized and the CUDA thread index; According to the target program and the corresponding relationship, converting the program corresponding to the subgraph to be optimized into a CUDA program; Compiling the CUDA program into optimized nodes by a CUDA compiler; The optimized nodes are used to replace the subgraph to be optimized in the neural network calculation graph to obtain an optimized neural network calculation graph; The step of converting the program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized comprises: According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained; The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

2. The method according to claim 1, characterized in that: Each node is configured with an index transformation matrix according to the corresponding operator type. The relationship between the index of the input data and the index of the output data of each node is represented by the corresponding index transformation matrix. The relationship between the index of the input data and the index of the output data of the subgraph to be optimized is represented by an index iteration matrix, and the index iteration matrix is ​​obtained according to the index transformation matrix corresponding to each node.

3. The method according to claim 2, characterized in that The converting the program corresponding to the subgraph to be optimized into a CUDA program according to the target program and the corresponding relationship includes: According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed; The input parameters of the kernel function are obtained to obtain the CUDA program.

4. The method according to claim 3, characterized in that The input parameters of the kernel function include the base address of the input data of the subgraph to be optimized, the base address of the output data, the dimension information of the input data, the dimension information of the output data and the elements of the index iteration matrix.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: Extracting a plurality of continuous target nodes in the neural network calculation graph; the target nodes are nodes whose indexes of input data and output data satisfy a linear transformation; A subgraph formed by the multiple target nodes is determined as the subgraph to be optimized.

6. A device for optimizing a neural network computation graph, characterized in that: The device comprises a CUDA program conversion module, a CUDA program optimization module and a subgraph replacement module; the CUDA program conversion module is configured as follows: Converting a program corresponding to a subgraph to be optimized into a target program represented by output data of the subgraph to be optimized; the subgraph to be optimized is a subgraph of the neural network computation graph; Establishing a correspondence between the index of the output data of the subgraph to be optimized and the CUDA thread index; According to the target program and the corresponding relationship, converting the program corresponding to the subgraph to be optimized into a CUDA program; The CUDA program optimization module is configured to: compile the CUDA program into optimized nodes through a CUDA compiler; The subgraph replacement module is configured to: use the optimized nodes to replace the subgraph to be optimized in the neural network calculation graph to obtain an optimized neural network calculation graph; In terms of converting the program corresponding to the subgraph to be optimized into a target program represented by the output data of the subgraph to be optimized, the CUDA program conversion module is specifically configured as follows: According to the relationship between the index of the input data and the index of the output data of each node in the subgraph to be optimized, the relationship between the index of the input data and the index of the output data of the subgraph to be optimized is obtained; The target program is obtained according to the relationship between the index of the input data and the index of the output data of the subgraph to be optimized and the relationship between the input data and the output data of each node.

7. The device according to claim 6, characterized in that Each node is configured with an index transformation matrix according to the corresponding operator type. The relationship between the index of the input data and the index of the output data of each node is represented by the corresponding index transformation matrix. The relationship between the index of the input data and the index of the output data of the subgraph to be optimized is represented by an index iteration matrix, and the index iteration matrix is ​​obtained according to the index transformation matrix corresponding to each node.

8. The device according to claim 7, characterized in that In terms of converting the program corresponding to the subgraph to be optimized into a CUDA program according to the target program and the corresponding relationship, the CUDA program conversion module is specifically configured as follows: According to the target program and the corresponding relationship, a kernel function represented by a CUDA thread is constructed; The input parameters of the kernel function are obtained to obtain the CUDA program.

9. The device according to claim 8, characterized in that The input parameters of the kernel function include the base address of the input data of the subgraph to be optimized, the base address of the output data, the dimension information of the input data, the dimension information of the output data and the elements of the index iteration matrix.

10. The device according to any one of claims 7 to 9, characterized in that: The device further includes a sub-image extraction module; the sub-image extraction module is configured to: Extracting a plurality of continuous target nodes in the neural network calculation graph; the target nodes are nodes whose indexes of input data and output data satisfy a linear transformation; A subgraph formed by the multiple target nodes is determined as the subgraph to be optimized.

11. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory is used to store one or more programs, and the programs include instructions for executing the method according to any one of claims 1 to 5.

12. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 5 is implemented.