Dynamic mode programming-oriented neural network accelerator memory optimization method and device
Through dynamic whole graph capture and neural network standard representation technology, comprehensive computing graphs are generated and collaboratively optimized, solving the problem of incomplete graph capture in dynamic programming mode, and improving the performance and memory management efficiency of neural network accelerators.
Patent Information
- Application Number
- CN202411321629.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-20
AI Technical Summary
The deep learning framework in dynamic programming mode has incomplete graph capture, which makes the system unable to optimize the global perspective, limiting the potential performance improvement of neural network accelerator memory optimization and resource allocation, and frequent computational graph recompilation increases memory overhead.
Dynamic whole graph capture technology and neural network standard representation method are used to generate reverse calculation diagrams through inverse topological order traversal, and collaborative optimization of forward and reverse calculation diagrams are carried out to generate optimized calculation diagrams, and the accelerator configuration is adjusted to improve performance.
Ensure the comprehensiveness and accuracy of the computing graph, improve the processing speed and memory capacity of the neural network accelerator, optimize system performance, and reduce the time and memory consumption of gradient calculations.
Smart Images
Figure CN119167981A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the optimization technology of a neural network accelerator, and relates to a neural network accelerator memory optimization method and device for dynamic mode programming. Background Art
[0002] Deep learning frameworks are divided into static mode and dynamic mode in terms of programming paradigm. Static mode (such as TensorFlow) emphasizes determining all calculation paths at compile time, which is conducive to performance optimization, but lacks flexibility; dynamic mode (such as PyTorch) allows dynamic adjustment of calculation graphs at runtime, which is highly flexible but difficult to optimize performance.
[0003] Deep learning frameworks under dynamic programming mode have problems such as incomplete graph capture. Existing solutions use a partial graph capture mechanism, which can improve computing performance, but the captured computational graph is often a specialized computational graph, and can only obtain part of the original computational graph. This makes it impossible for the system to optimize from a global perspective and unable to perform deep collaborative optimization. The lack of global perspective optimization limits the potential performance improvement of cross-level optimization such as neural network accelerator memory optimization and resource allocation. In addition, frequent computational graph recompilation increases the memory overhead of the neural network accelerator, and requires additional memory resources to manage and maintain multiple versions of computational graphs, which limits performance improvements. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention proposes a neural network accelerator memory optimization method and device for dynamic pattern programming. The method ensures the comprehensiveness and accuracy of the captured calculation graph, thereby optimizing the processing speed and memory capacity of the neural network accelerator and optimizing the system performance.
[0005] In order to achieve the above-mentioned object, the present invention provides a neural network accelerator memory optimization method for dynamic mode programming, comprising:
[0006] The required source code program written in the dynamic mode programming language is captured in its entirety to obtain the original computational graph, and the corresponding generative neural network standard representation is generated for the program fragments that can be captured in its entirety;
[0007] Using the standard representation of the neural network, the original computation graph is traversed in reverse topological order to generate a reverse computation graph;
[0008] The original computation graph and the reverse computation graph are collaboratively optimized to obtain an optimized computation graph;
[0009] The optimized computational graph is input into the neural network accelerator, and the configuration parameters of the neural network accelerator are adjusted according to the characteristics of the optimized computational graph and the hardware specifications of the neural network accelerator to execute the optimized computational graph.
[0010] In some embodiments, a source code program of a dynamic mode programming language is parsed to obtain bytecodes of a program to be executed;
[0011] The execution status of the bytecode is monitored and analyzed, the relationship between variables and operations is tracked during program execution, and the source code program is dynamically captured in the entire graph to obtain the original calculation graph.
[0012] In some embodiments, it also includes: for program fragments in the source code program that cannot be captured by the entire graph, executing a dynamic mode programming language interpreter is used as a fault-tolerant solution.
[0013] In some embodiments, using the neural network standard representation, before generating a reverse computation graph by traversing the original computation graph in reverse topological order, the original computation graph is standardized in advance to decompose complex neural network operations into basic atomic operations to form a consistent neural network standard representation.
[0014] In some embodiments, a linear computation graph is constructed for the original computation graph, where the linear computation graph represents a gradient computation process of the original computation graph;
[0015] Traverse the linear computation graph in reverse topological order, and generate reverse computation graph nodes for each node of the linear computation graph one by one;
[0016] Connect the nodes of the reverse calculation graph according to the dependencies contained in the original calculation graph to form a complete reverse calculation graph.
[0017] In some embodiments, for intermediate variables that need to be saved during the gradient calculation process of the original computation graph, representations of the intermediate variables are explicitly inserted and added into the linear computation graph.
[0018] In some embodiments, in the reverse computation graph, the input of each reverse computation graph node comes from the intermediate variables of the original computation graph and the outputs of other reverse computation graph nodes.
[0019] In some embodiments, it further comprises:
[0020] For each node of the reverse computation graph, starting from the output node, the gradient of each node is calculated in sequence until the gradients of all nodes are calculated.
[0021] In some embodiments, the neural network standard representation includes one or more combinations of operations representing calculations in a neural network, operations representing structured control flow information, and operations representing intermediate state management in a program.
[0022] In some embodiments, the optimized computation graph is analyzed to identify tasks that can be executed in parallel and tasks with data dependencies, and the tasks are partitioned to split the computation graph into multiple subtasks, each of which is executed in parallel at different stages of the pipeline;
[0023] According to the number and type of processing units of the neural network accelerator, the parallelism of the pipeline is adjusted, different subtasks of the computational graph are assigned to different processing units for execution, and the number of processing units for parallel execution of the neural network accelerator is updated.
[0024] Another aspect of the present invention further provides a neural network accelerator memory optimization device for dynamic mode programming, comprising:
[0025] A dynamic graph capture module is used to capture the entire graph of the required source code program written in the dynamic mode programming language to obtain the original computational graph, and to generate the corresponding generative neural network standard representation for the program fragment that can be captured in the entire graph;
[0026] An automatic differentiation module, for generating a reverse computation graph by traversing the original computation graph in reverse topological order using the standard representation of the neural network;
[0027] A collaborative optimization module, used to collaboratively optimize the original calculation graph and the reverse calculation graph to obtain an optimized calculation graph;
[0028] An execution module is used to input the optimized calculation graph into the neural network accelerator, adjust the configuration parameters of the neural network accelerator according to the characteristics of the optimized calculation graph and the hardware specifications of the neural network accelerator, and execute the optimized calculation graph.
[0029] It can be seen from the above scheme that the advantages of the present invention are:
[0030] The present invention discloses a method for optimizing the memory of a neural network accelerator for dynamic mode programming. By combining the dynamic whole-graph capture technology with the neural network standard representation method, the dynamic whole-graph capture technology can dynamically generate a complete calculation graph during program execution, capture the operations and structured control flow information in the calculation graph, and ensure the comprehensiveness and accuracy of the capture. The use of the neural network standard representation can decompose complex neural network operations into basic atomic operations, forming a consistent and easy-to-process representation form, thereby improving the efficiency of calculation graph management and operation. By automatically differentiating the captured original calculation graph, an accurate reverse calculation graph is generated, and the coordinated optimization of the forward calculation graph and the reverse calculation graph is realized. The optimized calculation graph supports automatic differentiation. By optimizing the gradient calculation process, the time and memory consumption of the gradient calculation can be reduced, thereby improving the data processing efficiency and response speed of the neural network accelerator and optimizing the memory structure. At the same time, the present invention further adjusts the configuration of the neural network accelerator according to the optimized calculation graph characteristics and the hardware specification characteristics of the neural network accelerator, thereby improving the internal performance optimization of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A schematic diagram of the overall process of a neural network accelerator memory optimization method for dynamic mode programming provided by an embodiment of the present invention is shown;
[0032] Figure 2 A schematic diagram of the structure of a neural network accelerator memory optimization device for dynamic mode programming is shown;
[0033] in:
[0034] 200: A neural network accelerator memory optimization device for dynamic mode programming;
[0035] 210: dynamic image capture module;
[0036] 220: automatic differentiation module;
[0037] 230: collaborative optimization module;
[0038] 240: Execution module. DETAILED DESCRIPTION
[0039] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.
[0040] See also Figure 1 , Figure 1 A schematic diagram of the overall process of a neural network accelerator memory optimization method for dynamic mode programming provided by an embodiment of the present invention is shown.
[0041] See also Figure 1As shown in , the neural network accelerator memory optimization method for dynamic mode programming specifically includes the following steps:
[0042] Step S1, capture the entire graph of the required source code program written in the dynamic mode programming language to obtain the original calculation graph, and generate the corresponding generative neural network standard representation for the program fragment that can be captured in the entire graph.
[0043] In some embodiments, the source code program of the dynamic mode programming language is parsed to obtain the bytecode of the program to be executed. Then, the execution state of the bytecode is monitored and analyzed, the relationship between variables and operations is tracked during the program execution, the source code program is dynamically captured in the entire graph to obtain the original computational graph, and the corresponding generative neural network standard representation is generated for the program fragment that can be captured in the entire graph.
[0044] It should be noted that the source code program encoded in dynamic mode under the deep learning framework can be a program written in different demand scenarios such as image classification, text classification, abnormal fault prediction, robot operation trajectory prediction, etc. The present invention does not limit its specific application field.
[0045] The neural network standard representation is a standard representation method specifically suitable for neural network computation graphs. The representation method aims to decompose the complex operations in the neural network into a series of basic, atomic-level operations, and standardize these operations to form a consistent and easy-to-process representation. The core abstraction of the neural network standard representation is operation (Operation, referred to as Op). The neural network standard representation uniformly represents the operations that may be involved in the neural network in the form of Op or a combination of Op. In some embodiments, the neural network standard representation mainly involves three types: operations ComputeOp representing operations in the neural network, operations ControlFlowOp representing structured control flow information, and operations StateOp representing intermediate state management in the program. By combining these Ops, the neural network standard representation can express various types of neural network programs, such as CNN, RNN, Transformer, etc. The neural network standard representation also retains the dependencies of the computational graph nodes, and can generate reverse computational graphs based on this information to achieve automatic differentiation.
[0046] In some embodiments, for program fragments in the source code program that cannot be captured by the entire graph, such as third-party libraries, program statements with side effects, etc., a dynamic mode programming language interpreter is used as a fault-tolerant solution.
[0047] In this embodiment, through the above-mentioned dynamic whole-graph capture, the computational graph that can be captured and the program code that cannot be captured in the source code program written in the dynamic mode programming language can be processed simultaneously. While ensuring the user's programming flexibility, it can efficiently capture the computational graph generated during the program execution process and retain its complete structural information and dependency relationships, which not only improves the comprehensiveness and accuracy of the capture, but also ensures the robustness in complex dynamic language environments.
[0048] Step S2: using the neural network standard representation, traversing the original computation graph in reverse topological order to generate a reverse computation graph.
[0049] In some embodiments, using the neural network standard representation, before the original calculation graph is traversed in reverse topological order to generate a reverse calculation graph, the original calculation graph is standardized in advance, and the complex neural network operation is decomposed into basic atomic operations, each atomic operation is an indivisible basic unit, forming a consistent neural network standard representation, ensuring that the standardized original calculation graph contains complete control flow information and dependencies. This representation method can accurately record the calculation details of each atomic operation, so that when performing gradient calculations, every tiny change can be tracked to ensure the accuracy and stability of the results. In addition, the representation method is also highly flexible and can support the expression of program constructs such as variable scope and structured control flow. This means that in the neural network model, complex control flow structures, such as loops and conditional branches, can also be effectively processed and expressed through this representation method, further enhancing the expressive power and scope of application of the calculation graph. Compared with the intermediate representation in the traditional compiler infrastructure (such as LLVM IR), on the one hand, the standard representation of the neural network encodes the high-level semantic information in the neural network, which is convenient for compilation optimization and scheduling at the computational graph level and for front-end docking with the deep learning framework; on the other hand, the standard representation method of the neural network can be gradually descended to the intermediate representation in the traditional compiler infrastructure through the intermediate representation, reusing the existing mature compilation optimization technology. Through this standardized representation, the neural network computational graph can be managed and operated more efficiently.
[0050] In some embodiments, a linear computation graph is further constructed for the original computation graph after the standardization process, and the linear computation graph represents the gradient computation process of the original computation graph. Then, the linear computation graph is traversed in reverse topological order, and reverse computation graph nodes are generated one by one for each node of the linear computation graph, and these nodes represent specific gradient computation operations. The reverse computation graph nodes are connected according to the dependency relationship contained in the original computation graph to form a complete reverse computation graph.
[0051] In this embodiment, the reverse topological order traversal ensures that before calculating the gradient of a node, all gradients that depend on the node have been calculated. For example, if node B depends on node A, the gradient of node A will be calculated before the gradient of node B.
[0052] In addition, in the linear computation graph, there may be some edges extending from the starting point of the original computation graph, which represent intermediate variables that need to be saved during the gradient calculation process. To manage the life cycle of these intermediate variables, in some embodiments, StateOp representing these intermediate variables is explicitly inserted into the linear computation graph to ensure that these intermediate variables are correctly saved and released during the gradient calculation process for subsequent gradient calculations.
[0053] In some embodiments, in the reverse computation graph, the input of each reverse computation graph node comes from the intermediate variables of the original computation graph and the outputs of other reverse computation graph nodes.
[0054] Finally, for each node of the reverse calculation graph, starting from the output node, the gradient of each node is calculated in turn until the gradient of all nodes is calculated. Through the calculation process of the above reverse calculation graph, efficient gradient calculation is achieved, ensuring that the training process of the complex neural network model can be carried out correctly.
[0055] In this embodiment, the neural network standard representation can be used to decompose complex neural network operations into basic atomic operations, forming a consistent and easy-to-process representation, which improves the efficiency of computational graph management and operation. By automatically differentiating the captured original computational graph, an accurate reverse computational graph is generated, and by optimizing the gradient calculation process in back propagation, the time and memory consumption of gradient calculation can be reduced, and the memory usage in the calculation process can be optimized, thereby improving the data processing efficiency and response speed of the neural network accelerator and optimizing the memory structure.
[0056] Step S3: collaboratively optimize the original calculation graph and the reverse calculation graph to obtain an optimized calculation graph.
[0057] In this embodiment, the correctness of the original calculation graph and the reverse calculation graph after the standardization processing is further checked, and the standardized original calculation graph and the reverse calculation graph are collaboratively optimized, that is, the collaborative optimization of the forward calculation graph and the reverse calculation graph is realized. For example, when possible, the original calculation graph and the reverse calculation graph reuse the memory already allocated in the neural network accelerator instead of frequently allocating and releasing new memory; or use a memory pool to manage the memory allocation of the original calculation graph and the reverse calculation graph in the neural network accelerator to reduce the overhead of allocating and releasing memory, etc., so as to improve the overall computing performance.
[0058] In this embodiment, the optimized computation graph supports automatic differentiation. By optimizing the gradient calculation process, the time and memory consumption of gradient calculation can be reduced, thereby improving the data processing efficiency and response speed of the neural network accelerator and optimizing the memory structure.
[0059] Step S4: input the optimized calculation graph into the neural network accelerator, adjust the configuration parameters of the neural network accelerator according to the characteristics of the optimized calculation graph and the hardware specifications of the neural network accelerator, and execute the optimized calculation graph.
[0060] In this embodiment, the configuration parameters of the neural network accelerator are further adjusted according to the characteristics of the optimized computation graph and the hardware specifications of the neural network accelerator.
[0061] For example, in some embodiments, according to the number of operations that can be executed in parallel in the optimized calculation graph and the hardware architecture characteristics of the neural network accelerator, by analyzing the optimized calculation graph, tasks that can be executed in parallel and tasks with data dependencies are identified, and task division is performed to split the calculation graph into multiple subtasks, each of which is executed in parallel at different stages of the pipeline. Further, according to the number and type of processing units of the neural network accelerator, the parallelism of the pipeline is adjusted. For an accelerator with multiple processing units, different subtasks of the calculation graph can be assigned to different processing units for execution, and the number of processing units of the neural network accelerator that execute in parallel is adjusted to achieve higher hardware parallelism and improve the overall processing speed.
[0062] In some embodiments, the cache structure of the neural network accelerator is adjusted according to the reuse of data in the optimized computational graph, and the cache capacity can be increased to reduce memory access latency for frequently accessed weights and feature graphs.
[0063] In addition, it should be noted that the neural network accelerator memory optimization method for dynamic mode programming disclosed in the present invention is not limited to being executed in the order of the above steps S1-S4. It can reorder, add or delete steps. As long as the expected results of the technical solution of the present invention can be achieved, the present invention is not limited here.
[0064] In summary, the present invention discloses a method for optimizing the memory of a neural network accelerator for dynamic mode programming. By combining the dynamic whole-graph capture technology with the neural network standard representation method, the dynamic whole-graph capture technology can dynamically generate a complete calculation graph during program execution, capture the operations and structured control flow information in the calculation graph, and ensure the comprehensiveness and accuracy of the capture. The use of the neural network standard representation can decompose complex neural network operations into basic atomic operations, forming a consistent and easy-to-process representation form, thereby improving the efficiency of calculation graph management and operation. By automatically differentiating the captured original calculation graph, an accurate reverse calculation graph is generated, and the forward calculation graph and the reverse calculation graph are collaboratively optimized. The optimized calculation graph supports automatic differentiation. By optimizing the gradient calculation process, the time and memory consumption of the gradient calculation can be reduced, thereby improving the data processing efficiency and response speed of the neural network accelerator and optimizing the memory structure. At the same time, the present invention further adjusts the configuration of the neural network accelerator according to the optimized calculation graph characteristics and the hardware specification characteristics of the neural network accelerator, thereby improving the internal performance optimization of the entire system.
[0065] In addition, corresponding to the aforementioned embodiment of the neural network accelerator memory optimization method for dynamic mode programming, the present invention also provides an embodiment of the neural network accelerator memory optimization device for dynamic mode programming.
[0066] See also Figure 2 , an embodiment of the present invention provides a neural network accelerator memory optimization device 200 for dynamic mode programming, the device at least comprising:
[0067] The dynamic graph capture module 210 is used to capture the entire graph of the required source code program written in the dynamic mode programming language to obtain the original calculation graph, and generate the corresponding generation neural network standard representation for the program fragment that can be captured in the entire graph.
[0068] The automatic differentiation module 220 is used to generate a reverse computation graph by traversing the original computation graph in reverse topological order using the neural network standard representation.
[0069] A collaborative optimization module 230, used to collaboratively optimize the original computation graph and the reverse computation graph to obtain an optimized computation graph;
[0070] The execution module 240 is used to input the optimized calculation graph into the neural network accelerator, adjust the configuration parameters of the neural network accelerator according to the characteristics of the optimized calculation graph and the hardware specifications of the neural network accelerator, and execute the optimized calculation graph.
[0071] Those skilled in the art can clearly understand that the description of the memory optimization method for a neural network accelerator for dynamic mode programming is convenient and concise, and the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0072] According to an embodiment of the present invention, the present invention also provides a machine-readable medium, which may be a tangible medium, which may contain or store a program for use by an instruction execution system, apparatus or device or for use in conjunction with an instruction execution system, apparatus or device, and when the computer program is executed by a processor, the steps of the above-mentioned neural network accelerator memory optimization method for dynamic mode programming are implemented.
[0073] In addition, the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0074] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A neural network accelerator memory optimization method for dynamic mode programming, characterized in that: Include: The required source code program written in the dynamic mode programming language is captured in its entirety to obtain the original computational graph, and the corresponding generative neural network standard representation is generated for the program fragments that can be captured in its entirety; Using the standard representation of the neural network, the original computation graph is traversed in reverse topological order to generate a reverse computation graph; The original computation graph and the reverse computation graph are collaboratively optimized to obtain an optimized computation graph; The optimized computational graph is input into the neural network accelerator, and the configuration parameters of the neural network accelerator are adjusted according to the characteristics of the optimized computational graph and the hardware specifications of the neural network accelerator to execute the optimized computational graph.
2. The method according to claim 1, characterized in that Parsing a source code program of a dynamic mode programming language to obtain bytecodes of a program to be executed; The execution status of the bytecode is monitored and analyzed, the relationship between variables and operations is tracked during program execution, and the source code program is dynamically captured in the entire graph to obtain the original calculation graph.
3. The method according to claim 1, characterized in that It also includes: for program fragments in the source code program that cannot be captured by the whole graph, executing a dynamic mode programming language interpreter is used as a fault-tolerant solution.
4. The method according to claim 1, characterized in that By using the standard representation of the neural network, the original computational graph is standardized before generating a reverse computational graph by traversing the original computational graph in reverse topological order, and the complex neural network calculations are decomposed into basic atomic operations to form a consistent standard representation of the neural network.
5. The method according to claim 1, characterized in that Constructing a linear computation graph for the original computation graph, where the linear computation graph represents a gradient computation process of the original computation graph; Traverse the linear computation graph in reverse topological order, and generate reverse computation graph nodes for each node of the linear computation graph one by one; Connect the nodes of the reverse calculation graph according to the dependencies contained in the original calculation graph to form a complete reverse calculation graph.
6. The method according to claim 5, characterized in that For the intermediate variables that need to be saved in the gradient calculation process of the original calculation graph, the representation of the intermediate variables is explicitly inserted into the linear calculation graph.
7. The method according to claim 6, characterized in that In the reverse computation graph, the input of each reverse computation graph node comes from the intermediate variables of the original computation graph and the output of other reverse computation graph nodes.
8. The method according to claim 5, characterized in that Also includes: For each node of the reverse computation graph, starting from the output node, the gradient of each node is calculated in sequence until the gradients of all nodes are calculated.
9. The method according to claim 1, characterized in that: The neural network standard representation includes one or more combinations of operations representing calculations in a neural network, operations representing structured control flow information, and operations representing intermediate state management in a program.
10. The method according to claim 1, characterized in that Analyze the optimized computation graph, identify tasks that can be executed in parallel and tasks with data dependencies, divide the tasks, split the computation graph into multiple subtasks, and execute each subtask in parallel at different stages of the pipeline; According to the number and type of processing units of the neural network accelerator, the parallelism of the pipeline is adjusted, different subtasks of the computational graph are assigned to different processing units for execution, and the number of processing units for parallel execution of the neural network accelerator is updated.
11. A neural network accelerator memory optimization device for dynamic mode programming, characterized in that: Include: A dynamic graph capture module is used to capture the entire graph of the required source code program written in the dynamic mode programming language to obtain the original computational graph, and to generate the corresponding generative neural network standard representation for the program fragment that can be captured in the entire graph; An automatic differentiation module, for generating a reverse computation graph by traversing the original computation graph in reverse topological order using the standard representation of the neural network; A collaborative optimization module, used to collaboratively optimize the original calculation graph and the reverse calculation graph to obtain an optimized calculation graph; An execution module is used to input the optimized calculation graph into the neural network accelerator, adjust the configuration parameters of the neural network accelerator according to the characteristics of the optimized calculation graph and the hardware specifications of the neural network accelerator, and execute the optimized calculation graph.