A method, apparatus and device for optimizing a computational graph

Abstract

This application provides a computation graph optimization method, apparatus, and device, relating to the field of artificial intelligence technology. The method can obtain a second computation graph from a first computation graph based on parameters and data dependencies between nodes in the first computation graph; then, the second computation graph can be merged with the first computation graph to obtain a new computation graph; and the operators in this new computation graph are fused to obtain an optimized computation graph. Finally, the optimized computation graph can be executed. Thus, by optimizing the computation graph through a combination of recomputation and operator fusion, it significantly reduces memory usage without introducing large recomputation overhead, solving the problem of networks with one or more extremely large tensors being unable to execute.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a computational graph optimization method, apparatus, and device. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Currently, with the continuous development of computer technology, AI networks have been widely applied. Furthermore, AI networks are becoming increasingly complex, with their network models showing a trend of increasing size. For example, the number of network layers, parameters, and datasets in AI networks is growing, leading to increasingly higher memory consumption for these models. Since current computer hardware has relatively limited memory, commonly 16 gigabytes (GB) or 32GB, current computer hardware will struggle to support the increasing memory consumption of AI network models. Therefore, reducing the memory consumption of AI network models is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a computation graph optimization method, apparatus, device, computer storage medium, computer program product, and chip. By combining recomputation with operator fusion, the computation graph is optimized, which significantly reduces memory usage without introducing large recomputation overhead, and solves the problem that networks with one or more extremely large tensors cannot be executed.

[0005] In a first aspect, this application provides a computation graph optimization method, comprising: obtaining a second computation graph from the first computation graph based on parameters and data dependencies between nodes in the first computation graph, wherein the parameters include one or more of the following: operator fusion rules, a memory threshold for the memory occupied by tensors output by a single node in the first computation graph, a peak memory threshold corresponding to the first computation graph, and a data mutation threshold corresponding to a node in the first computation graph; the peak memory threshold is the threshold for the memory occupied by all tensors required at the moment a node is executed during the execution of the computation graph, and the data mutation threshold is the threshold for at least one node in the first computation graph to generate a data mutation; merging the second computation graph with the first computation graph to obtain a third computation graph, wherein in the third computation graph, a first node outputting a first tensor and a second node inputting a first tensor have a first directed edge, and a third node outputting a second tensor and a fourth node inputting a second tensor have a second directed edge, the first directed edge pointing from the first node to the second node, and the second directed edge pointing from the third node to the fourth node, the first node and the fourth node correspond to nodes in the first computation graph, and the second node and the third node correspond to nodes in the second computation graph; fusing operators in the third computation graph to obtain a fourth computation graph; and executing the fourth computation graph. In this way, based on the parameters, a second computation graph that needs to be recomputed is selected from the first computation graph, and the second computation graph is merged with the first computation graph to obtain a new computation graph. Operator fusion is then performed on the new computation graph, and the fused computation graph is executed. Thus, by combining recomputation and operator fusion, the computation graph is optimized, which significantly reduces memory usage without introducing large recomputation overhead, and solves the problem that networks with one or more very large tensors cannot be executed.

[0006] In one possible implementation, a second computational graph is obtained from the first computational graph based on parameters and data dependencies between nodes in the first computational graph. Specifically, this includes: obtaining N sets of first nodes from the producers included in the first computational graph, where N is a positive integer greater than or equal to 1. Each set of first nodes includes at least one node, and each node in a set is directly or indirectly related to a tensor output by a producer in the first computational graph, and also directly or indirectly related to at least one tensor input by a producer in the first computational graph. Each set of first nodes is then recomputed to obtain N recomputable subgraphs, which together constitute the second computational graph. In this way, the second computational graph can be obtained from the first computational graph based on the data dependencies between nodes and parameters in the first computational graph.

[0007] In one possible implementation, the parameter is a memory threshold; the memory occupied by the output tensor corresponding to the first node set exceeds the memory threshold. This automatically selects the node set corresponding to the output tensor that occupies excessive memory as the required node set, thus solving the problem of memory consumption by a single operator and / or tensor.

[0008] In one possible implementation, the parameter is a peak memory threshold; there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold. This automatically identifies the node set with excessively high peak memory usage as the required node set, thus solving the problem of memory consumption by individual operators and / or tensors.

[0009] In one possible implementation, the parameter is a data mutation threshold; the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This automatically selects the node set with data mutations as the required node set, thus eliminating the problem of excessive memory usage caused by data mutations.

[0010] In one possible implementation, the parameter is the operator fusion rule; the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows the desired node set to be selected using the operator fusion rule.

[0011] In one possible implementation, the parameters are a memory threshold and a peak memory threshold; there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; and / or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold. This allows for the selection of the desired node set by combining the memory threshold and the peak memory threshold, improving the efficiency and accuracy of the selection process.

[0012] In one possible implementation, the parameters are a memory threshold and a data mutation threshold; the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, and / or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This allows for the selection of the desired node set by combining the memory threshold and the data mutation threshold, improving the efficiency and accuracy of the selection process.

[0013] In one possible implementation, the parameters are a memory threshold and an operator fusion rule. The first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows for the selection of the desired node set by combining the memory threshold and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0014] In one possible implementation, the parameters are a peak memory threshold and a data mutation threshold; there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; and / or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This allows for the selection of the desired node set by combining the peak memory threshold and the data mutation threshold, improving the efficiency and accuracy of the selection process.

[0015] In one possible implementation, the parameters are a memory threshold and an operator fusion rule. The first node set meets one or more of the following conditions: there is an indirect path between the producer and consumer corresponding to the first node set, and the peak memory usage of each node between the producer and consumer of the first node set is higher than the peak memory threshold; or, the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer of the first node set. This allows for the selection of the desired node set by combining the peak memory threshold and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0016] In one possible implementation, the parameters are a data mutation threshold and an operator fusion rule. The first node set meets one or more of the following conditions: the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows for the selection of the desired node set by combining the data mutation threshold and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0017] In one possible implementation, the parameters are a memory threshold, a peak memory threshold, and a data mutation threshold. The first node set meets one or more of the following conditions: there is no indirect path between the producer and consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, there is an indirect path between the producer and consumer corresponding to the first node set, and the peak memory usage of each node between the producer and consumer corresponding to the first node set is higher than the peak memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This allows for the selection of the desired node set by combining the memory threshold, peak memory threshold, and data mutation threshold, improving the efficiency and accuracy of the selection process.

[0018] In one possible implementation, the parameters are a memory threshold, a peak memory threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node during execution between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the first node set and its corresponding consumers can be fused; or, the first node set and its corresponding consumers cannot be fused, and there is an indirect path between the producers and consumers corresponding to the first node set. This allows for the selection of the desired node set through a combination of the memory threshold, peak memory threshold, and operator fusion rule, improving the efficiency and accuracy of the selection process.

[0019] In one possible implementation, the parameters are a memory threshold, a data mutation threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows for the selection of the desired node set through a combination of the memory threshold, data mutation threshold, and operator fusion rule, improving the efficiency and accuracy of the selection process.

[0020] In one possible implementation, the parameters are a peak memory threshold, a data mutation threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set can be fused with its corresponding consumers; or, the first node set cannot be fused with its corresponding consumers, and there is an indirect path between the producers and consumers corresponding to the first node set. This allows for the selection of the desired node set by combining the peak memory threshold, the data mutation threshold, and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0021] In one possible implementation, the parameters are a memory threshold, a peak memory threshold, a data mutation threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory of each node executing between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set and its corresponding consumers can be fused; or, the first node set and its corresponding consumers cannot be fused, and there is an indirect path between the producers and consumers corresponding to the first node set. This allows for the selection of the desired node set through a combination of the memory threshold, peak memory threshold, data mutation threshold, and operator fusion rule, improving the efficiency and accuracy of the selection process.

[0022] In one possible implementation, based on parameters and data dependencies between nodes in the first computation graph, at least one set of first nodes is obtained from the producers contained in the first computation graph, specifically including:

[0023] Based on the data dependencies between nodes in the first computation graph, a first list is obtained, which includes the correspondence between producers and consumers in the first computation graph. Based on the tensors output by each producer in the first list, a candidate node set is obtained, which includes at least the first node set. Specifically, for any producer in the first list, the set of nodes directly and indirectly related to the tensors output by that producer is considered as a node set. Based on parameters, the first node set is selected from the candidate node set. Thus, the first node set can be obtained from the producers in the first computation graph.

[0024] In one possible implementation, each first node set is recomputed to obtain N recomputable subgraphs. Specifically, this includes: for any node set among the N first node sets, copying the producer subgraph corresponding to any node set; deleting nodes in the producer subgraph other than those contained in any node set, and deleting edges in the producer subgraph that are unrelated to any node set, to obtain the recomputable subgraph corresponding to any node set.

[0025] In one possible implementation, merging the second computation graph with the first computation graph to obtain a third computation graph specifically includes: constructing directed edges between nodes in each of the N recomputation subgraphs that input the first target tensor and nodes in the first computation graph that output the first target tensor; and constructing directed edges between nodes in each of the N recomputation subgraphs that output the second target tensor and nodes in the first computation graph that input the second target tensor, to obtain the third computation graph.

[0026] In one possible implementation, before obtaining the second computation graph requiring recomputation from the first computation graph based on parameters and data dependencies between nodes in the first computation graph, the method further includes fusing operators in the first computation graph. This improves the performance of operators in the computation graph; additionally, by fusing operators in the first step, the number of operators can be reduced, thus reducing the overhead of analyzing operators during subsequent recomputation; furthermore, it can expand the scope of operator analysis during subsequent recomputation, improving the effectiveness of recomputation.

[0027] In a second aspect, this application provides a computational graph optimization apparatus, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

[0028] Thirdly, this application provides an apparatus comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method provided in the first aspect.

[0029] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method provided in the first aspect.

[0030] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method provided in the first aspect.

[0031] In a sixth aspect, this application provides a chip including at least one processor and an interface; the interface is used to provide program instructions or data to at least one processor; the at least one processor is used to execute program line instructions to implement the method provided in the first aspect.

[0032] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram illustrating the training process of a network model for an AI network provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram illustrating the training process of another AI network model provided in an embodiment of this application;

[0036] Figure 4 This is a comparative schematic diagram of operator fusion before and after an embodiment of this application;

[0037] Figure 5 This is a schematic diagram comparing peak memory usage before and after operator fusion, provided in an embodiment of this application.

[0038] Figure 6 This is a schematic diagram of a graph computing fusion process under an AI open-source computing framework provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of an operator fusion process provided in an embodiment of this application;

[0040] Figure 8This is a schematic diagram illustrating a topological sorting of operators in a computation graph, provided in an embodiment of this application.

[0041] Figure 9 This is a schematic diagram illustrating a step of filtering a set of recomputed nodes using a memory-based method, as provided in an embodiment of this application.

[0042] Figure 10 This is a schematic diagram illustrating another step in filtering the set of recomputed nodes using a memory-based method, as provided in an embodiment of this application.

[0043] Figure 11 This is a schematic diagram of a computation graph provided in an embodiment of this application;

[0044] Figure 12 This is a schematic diagram comparing peak memory usage before and after recalculating the computation graph, provided in an embodiment of this application.

[0045] Figure 13 This is a schematic diagram illustrating a step of filtering a set of recomputable nodes using operator fusion rules, as provided in an embodiment of this application.

[0046] Figure 14 This is a schematic diagram illustrating a process for determining a recalculation graph provided in an embodiment of this application;

[0047] Figure 15 This is a schematic diagram illustrating the process of merging a determined recomputation graph with a computation graph obtained after performing an operator fusion on the original computation graph, as provided in an embodiment of this application.

[0048] Figure 16 This is a schematic diagram of an operator fusion process provided in an embodiment of this application;

[0049] Figure 17 This is a schematic diagram comparing the memory size required before and after optimizing the computation graph using operator fusion and recomputation, as provided in an embodiment of this application.

[0050] Figure 18 This is a flowchart illustrating a computational graph optimization method provided in an embodiment of this application;

[0051] Figure 19 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0052] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0053] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0056] To facilitate understanding, the technical terms used in this solution will be introduced first.

[0057] (1) Graphical-Computational Integration

[0058] Graph-Computation Fusion is a unique network performance optimization technology of MindSpore. It automatically analyzes and optimizes the logic of the existing network computation graph, and, combined with the target hardware capabilities, performs computational simplification and substitution, operator splitting and fusion, and operator-specific compilation to improve the utilization of device computing resources and achieve overall network performance optimization. Compared to traditional optimization techniques, Graph-Computation Fusion has unique advantages such as multi-operator cross-boundary joint optimization, cross-layer collaboration with MindAKG (an operator compiler based on Polyhedral), and just-in-time compilation. Furthermore, once the user enables the corresponding configuration, the entire optimization process is completed automatically, requiring no additional awareness from network developers, allowing users to focus on network algorithm implementation.

[0059] (2) Tensor

[0060] A tensor is a data structure used in AI frameworks to represent data. It is typically composed of related attributes such as shape (representing the size and dimensions of the data), data type (such as FLAOT32, int, etc.), and memory address.

[0061] (3) Operators

[0062] In the field of deep learning, an operator generally refers to a computational unit that can perform simple or complex calculations such as addition, subtraction, multiplication, and division. On acceleration hardware, these calculations are usually performed in units of operators.

[0063] (4) Memory reuse

[0064] Memory reuse is an optimization technique. In deep learning networks, physical memory can be reused by pointing tensors from different layers to the same address, thereby reducing memory usage.

[0065] (5) Calculation graph

[0066] A computation graph is a directed graph that represents computational functions, with operators as nodes. A computation graph is primarily represented by nodes and edges. Nodes in a computation graph can represent operators. An edge between two operators in a computation graph can represent the relationship between the two operators. Edges can have a direction, which can represent the flow of data between operators. For example, when an edge between operator a and operator b in a computation graph points from operator a to operator b, then the tensor output by operator a becomes the input of operator b. In AI frameworks, the computational functions represented by the computation graph sequentially call the operator nodes in the directed graph on the input tensor and obtain the final output tensor.

[0067] (6) Peak memory

[0068] Peak memory refers to the total memory occupied by all tensors required at any given moment during the execution of a particular operator in the computation graph. For example, please refer to [link to example]. Figure 5 In (A), at the moment when operator D is executed, the tensor output by operator A, the tensor output by operator C, and the tensor to be output by operator D are all the tensors that need to be used. Therefore, the peak memory at this time is T0+T1+T2; among them, the input tensor of operator B and the output tensor of operator C share the same memory.

[0069] (7) Producers

[0070] A producer is a computational subgraph that generates data between two computational subgraphs that have data dependencies; in other words, it is a data-generating operator. For example, see [link to example]. Figure 4 Taking operators O1 and O2 as examples, operators O1 and O2 have a data dependency relationship. The output tensor of operator O1 is the input tensor of operator O2. Therefore, operator O1 can be called a producer.

[0071] (8) Consumers

[0072] Consumers are computational subgraphs that use data within two computational subgraphs that have data dependencies; that is, they are operators that use data. For example, see [link to example]. Figure 4Taking operators O1 and O2 as examples, operators O1 and O2 have a data dependency relationship. The output tensor of operator O1 is the input tensor of operator O2. Therefore, operator O2 can be called a consumer.

[0073] (9) Node set

[0074] A node set refers to a collection of one or more nodes. Each node in a node set is directly or indirectly related to a tensor output by a producer in the computation graph, and is also directly or indirectly related to at least one tensor input by that producer. Furthermore, it can be understood that when a node set includes multiple nodes, the computation graph composed of these nodes can obtain at least one tensor from the outside and output a tensor to the outside. That is, at least one node among these multiple nodes can obtain tensors input from the outside, and only one node can output tensors to the outside, while the tensors output by the other nodes can only be used internally within the computation graph.

[0075] For example, such as Figure 11 As shown, Gather, Scatter, and Div can form a set of nodes. In the computation graph formed by these three nodes, Gather, Scatter, and Div are sequentially connected, with an edge between Gather and Scatter, and an edge between Scatter and Div. Each of these three nodes is directly or indirectly related to the tensor B output by its corresponding producer FuseOp1, and also directly or indirectly related to the tensor A input by producer FuseOp1. In other words, the computation graph formed by these three nodes can obtain a tensor A from the outside through Gather and output a tensor B to the outside through Div. However, Gather and Scatter cannot output tensors outside the computation graph; their output tensors can only be used within the computation graph. Figure 14 As shown, in example 14(B), Gather can form a set of nodes, and the computation graph corresponding to this set of nodes can be as follows: Figure 14 As shown in (C), the computation graph can obtain a tensor A from the outside and output a tensor B to the outside.

[0076] (10) Data dependencies between nodes

[0077] A data dependency exists between two nodes when the data required by one node is provided by another node. For example, a data dependency exists between two nodes when the output tensor of one node is the input tensor of another node.

[0078] (11) Data mutation threshold

[0079] The data mutation threshold refers to the threshold at which at least one node in the computation graph generates a data mutation. Taking a single node as an example, the data mutation threshold is the threshold value (e.g., difference or ratio) between the memory occupied by the node's output tensor and the memory occupied by the node's input tensor. Taking multiple nodes as an example, these multiple nodes can form the node set described above. The data mutation threshold is then the threshold value (e.g., difference or ratio) between the memory occupied by the output tensor corresponding to this node set and the memory occupied by at least one input tensor corresponding to this node set. Here, the output tensor corresponding to the node set refers to the tensor output by the computation graph composed of the nodes in the node set to the outside, and the input tensor corresponding to the node set refers to the tensor obtained by the computation graph composed of the nodes in the node set from the outside.

[0080] The technical solutions in the embodiments of this application will be described next.

[0081] For example, Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0082] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0083] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0084] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0085] (1) Infrastructure:

[0086] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0087] (2) Data

[0088] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0089] (3) Data processing

[0090] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0091] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0092] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0093] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0094] (4) General ability

[0095] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0096] (5) Smart Products and Industry Applications

[0097] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0098] It should be noted that the operator fusion process involved in this application can be located in the data processing stage in (3) above.

[0099] For example, Figure 2 The training process of an AI network model is illustrated. This training process can include forward computation and backward computation. For example... Figure 1 As shown, the part to the left of the dashed line x represents the forward computation process, and the part to the right of the dashed line x represents the backward computation process. Each square in the diagram represents a tensor. Figure 1 In this context, tensors located on the same row share the same memory address, for example: tensor A and tensor A2. b Reusing a memory address, tensor B and tensor B b Reusing a memory address, ..., tensor F and tensor F b Reusing a memory address, etc. Furthermore, Figure 1 In the middle, tensor A b Tensor B b Tensor C b Tensor D b Tensor E b Tensor F b These are the inverse operator data corresponding to tensors A, B, C, D, E, and F, respectively. Figure 1 In the training process of a network model, forward computation can be performed first, followed by backward computation. The tensors obtained from the forward computation are stored in the memory of the corresponding computing device. During the backward computation, tensor A... b The computation depends on tensors In and B. b Tensor B b The computation depends on tensor A and tensor C. b Tensor C b The computation depends on tensor B and tensor D. b Tensor D b The computation depends on tensor C and tensor E b And so on. During the backward computation, after a tensor is calculated, the memory of the tensor corresponding to that tensor obtained in the forward computation is released. For example, after calculating tensor B... b Then, the memory occupied by tensor A is released, so that the tensor A calculated subsequently can be used for other operations. b This allows the memory occupied by tensor A to be reused. It is evident that in... Figure 1 During the training process of the network model shown, the data obtained from the forward computation will occupy memory for a long time, resulting in excessive memory consumption of the network model.

[0100] Generally, in Figure 1 The memory usage issue can be addressed using either recomputation or operator fusion. Specifically, when using recomputation, it is... Figure 1 China adopted a time-for-space approach to release resources ahead of schedule. Figure 1 The memory occupied by the partial result obtained from the forward computation is reduced, and this partial result is recalculated in subsequent backward computations, thereby shortening the lifespan of this data and thus reducing memory usage. Specifically, for example... Figure 3 As shown, tensor A obtained from forward computation can be used to obtain tensor B from backward computation. bPreviously, it continuously occupied memory. Tensor D obtained from forward computation can be used to obtain tensor E from backward computation. b Previously, memory was continuously occupied. The memory occupied by the four tensors B, C, E, and F obtained from the forward computation can be released immediately after use during the forward computation, and the computation processes corresponding to these four tensors are re-executed during the reverse computation. Specifically, the memory occupied by tensor B can be released after tensor C is obtained from the forward computation, the memory occupied by tensor C can be released after tensor D is obtained from the forward computation, and the memory occupied by tensor D can be released when tensor D is calculated backward. b When the tensor C is recalculated, it can be recalculated. b In this case, tensor B can be recalculated. This recalculation method reduces memory usage. See further... Figure 1 If in Figure 1 Each square occupies one memory space, and the forward computation ends when the tensor F is obtained. Therefore, the data obtained after the forward computation occupies a total of 7 memory spaces; while... Figure 3 In the above, since the memory occupied by tensors B, C, E, and F is immediately released after they are used during the forward computation, these four tensors can be considered to occupy no memory. Figure 3 The data obtained after the forward computation occupies a total of 3 memory units. This demonstrates that recomputation can reduce memory usage.

[0101] However, when using recomputation, it's often necessary to manually specify recomputation points or automatically calculate them. Manual recomputation often requires users to identify recomputation split points, demanding high user skills and requiring separate settings for different models and hardware backends, resulting in low efficiency. Automatic recomputation either has long solution times (on the order of hours, exceeding most training tasks) or requires runtime statistics, making it impossible to integrate with operators. Furthermore, recomputation can only handle data dependencies between forward and backward rows, not between forward-forward or backward-backward rows, limiting its application scenarios. Additionally, recomputation cannot solve the memory consumption problem of individual operators and single extremely large tensors. For example, see [further details omitted]. Figure 3 If the data volume of tensor A is too large, it will occupy a large amount of memory, which will result in a situation where memory is continuously and significantly occupied.

[0102] When using operator fusion, network performance is improved by combining multiple adjacent operators into a single operator, while also reducing memory usage. Specifically, for example... Figure 4 As shown, Figure 4(A) represents the calculation method before fusion, where tensor A is input into operator O1 to obtain tensor B, tensor B is input into operator O2 to obtain tensor C, and tensor C is input into operator O3 to obtain tensor D. By fusing operators O1, O2, and O3 using pre-defined fusion rules, these three operators can be merged into one operator. At this point, the process of obtaining tensor D from tensor A is transformed into... Figure 4 As shown in (B). In Figure 4 In (B), tensor A is input into the fused operator (O1+O2+O3) to obtain tensor D. Figure 4 Comparing (A) and (B), it can be seen that, Figure 4 There are 4 tensors in (A) that require memory, while Figure 4 Two tensors in (B) require memory. Therefore, using operator fusion can reduce memory usage.

[0103] However, limited by internal storage size and hardware processing power, operator fusion can only process operators in adjacent spaces, limiting its scope and scale. Furthermore, a significant amount of computation exists between the forward and backward operators, making direct fusion impossible; the results of the forward computation still need to be temporarily stored in memory. Additionally, in some multi-input scenarios, operator fusion extends the lifetime of different tensors to the entire fusion operator, actually increasing memory usage. For example, ... Figure 5 As shown, Figure 5 (A) represents the execution order of each operator before operator fusion, the lifetime of the input and output tensors of each operator, and the peak memory usage of each operator during execution. Figure 5 (B) represents the execution order of each operator after operator fusion, the lifetime of the input and output tensors of each operator, and the peak memory usage of each operator during execution. Among these, in... Figure 5 In the example, operators A, B, and C can be merged into operator E. Figure 5 In (A), the tensor output by operator B shares the same memory as the tensor input by operator A, the tensor output by operator C shares the same memory as the tensor input by operator B, and the tensor output by operator D shares the same memory as the tensor input by operator C. Figure 5 In (B), the tensor output by operator D and a tensor input by operator E can reuse one memory space. When operator C executes, since this is equivalent to operator E executing, memory spaces T0, T1, and T2 cannot be reused at this time; instead, another memory space, T3, must be allocated. Figure 5 From (A), we know that before operator fusion, the peak memory when using this computation graph is (T0+T1+T2), which is derived from... Figure 5As shown in (B), the peak memory when using the computation graph after operator fusion is (T0+T1+T2+T3). It is evident that the peak memory after operator fusion is significantly larger than the peak memory before operator fusion.

[0104] As the above analysis shows, although recomputation or operator fusion can solve some of the memory usage problems, they still cannot simultaneously solve the problems of network failure caused by high peak memory usage due to long-lived tensors that span long path dependencies, and network failure caused by a single operator or a single tensor exceeding memory limits.

[0105] In view of this, this application provides a computation graph optimization method. This method mainly involves obtaining a recomputation graph requiring recomputation from the computation graph to be optimized, merging the obtained recomputation graph with the computation graph to be optimized to generate a new computation graph, and finally performing operator fusion on the new computation graph. Thus, by combining recomputation and operator fusion, this method significantly reduces memory usage without introducing large recomputation overhead (including compile-time and runtime), solving the problem of networks with one or more extremely large tensors being unable to execute.

[0106] For example, Figure 6 A schematic diagram of a graph-computation fusion process under an open-source AI computing framework is shown. Figure 6 The open-source AI computing framework shown can be MindSpore. Figure 6As shown, this open-source AI computing framework can include a front-end representation layer (not shown in the figure), a MindSpore layer, and a MindSpore Auto Kernel Generator layer (MindSpore AGK layer). The front-end representation layer provides users with at least one application programming interface (API). When using this open-source AI computing framework, users can select their desired network model 401 in the front-end representation layer and configure various data through it. Then, the network model and the user-configured MindSpore Expressio (ME) front-end representation can be automatically generated in the MindSpore layer. Next, a computational graph can be automatically constructed in the MindSpore layer based on the user-configured ME front-end representation. Common optimizations can then be performed on the computational graph, such as eliminating unnecessary code and optimizing recurring data. Then, back-end optimizations are performed on the computational graph, such as selecting more suitable representations for different hardware characteristics. After back-end optimization, an optimized computational graph is generated. The process of obtaining the optimized computational graph can be understood as a graph-computation fusion process. The computation graph optimized through graph-computation fusion can contain several operator definition descriptions, which explain the computation process of each operator in the computation graph. In the MindSpore AGK layer, data can be passed to Mind AKG in JSON format for compilation and optimization, and a code generator can be invoked to generate backend hardware code and compile it into a backend kernel. The backend kernel can be understood as the hardware representation of the operators, such as the corresponding binary files. After the layer and operator compilation are completed, the MindSpore layer can call the kernel according to the order of the computation graph and execute the corresponding computation graph.

[0107] This application primarily focuses on the backend optimization portion of the MindSpore layer. The backend optimization within the MindSpore layer can reduce the network model's memory consumption through recomputation and operator fusion, thereby providing performance and memory optimization for different types of hardware backends (e.g., graphics processing units, neural network processors, central processing units, etc.).

[0108] The following detailed description of the computational graph optimization scheme provided in this application is based on the MindSporeAI open-source computing framework described above and in conjunction with the accompanying drawings. It is understood that the MindSporeAI open-source computing framework described in the embodiments of this application is merely illustrative, and the computational graph optimization method provided in the embodiments of this application is not limited to this framework and can also be applied to other frameworks. When applied to other frameworks, it is still within the scope of protection of this application.

[0109] For example, this computational graph optimization scheme can, but is not limited to, run on any device, equipment, platform, or cluster of devices with computing and processing capabilities. The scheme mainly includes two steps: setting up a configuration interface and computational graph optimization. Setting up the configuration interface allows users to configure necessary parameters for use during the computational graph optimization process. Computational graph optimization is primarily a process combining operator fusion and recomputation, as detailed below.

[0110] (I) Setting up configuration items interface

[0111] MindSpore offers configurable Context and Graph Fusion modules. The Context module receives global user settings for the computation process. The Graph Fusion module can, but is not limited to, performing graph-to-calculation fusion on the computation graph and operators. This module can, but is not limited to, be configured within MindSpore's AI Compiler module, which is responsible for compiling and optimizing the computation graph and operators.

[0112] In this embodiment, two configuration item interfaces may be added to the Context module, but are not limited to: one configuration item interface for configuring memory thresholds and / or peak memory thresholds, and the other configuration item interface for configuring data thresholds. Users can set the values ​​of these two configuration items according to their actual network and dataset scenarios. When the user does not set them, the system default values ​​can be used, or they can be automatically generated based on the hardware environment. For example, the video memory size of the graphics processing unit (GPU) can be used as the memory threshold and / or peak memory threshold, but is not limited to.

[0113] It is understandable that the configuration settings interface can be set in real time or in advance, depending on the actual situation, and no restrictions are imposed here.

[0114] (II) Optimization of the computational graph

[0115] The computational graph optimization process primarily involves a combination of operator fusion and recomputation. This process can be executed, but is not limited to, within the graph fusion module configured in MindSpore. This graph fusion module can read the memory threshold and / or peak memory threshold, and data threshold configured by the user in the Context module. The computational graph optimization process mainly includes primary operator fusion, recomputation, and secondary operator fusion, as detailed below.

[0116] (1) First-order operator fusion

[0117] When optimizing a computation graph, operator fusion can be performed first. For example, but not limited to, based on pre-defined operator fusion rules (e.g., element-wise + element-wise operator fusion rules, element-wise + reduction operator fusion rules, segment + element-wise operator fusion rules, etc.), and according to the type of operators in the computation graph (e.g., computationally intensive, memory-intensive, etc.) and the backend characteristics of the hardware, the computation graph can be divided into multiple computation subgraphs. Each computation subgraph can correspond to a fused operator (hereinafter referred to as the "fusion operator"), thus obtaining the optimized computation graph, i.e., the first-stage fused subgraph. This improves the performance of operators in the computation graph. Furthermore, the first operator fusion can reduce the number of operators, thereby reducing the overhead of operator analysis during subsequent recomputation. Additionally, it can expand the scope of operator analysis during subsequent recomputation, improving the recomputation effect. For example, please refer to... Figure 11 Assuming that before operator fusion, operator Div does not experience data mutation, and after operator fusion, the output tensor B corresponding to FuseOp1 is much larger than its corresponding input tensor A, then before operator fusion, it is difficult to use data mutation to treat operator Div as a set of recomputation nodes. However, after operator fusion, the fusion operator FuseOp1 corresponding to operator Div experiences data mutation. Therefore, the set of operators related to the output tensor B of operator Div can be used as the set of recomputation nodes. In this way, operator fusion expands the scope of operator analysis during the recomputation process.

[0118] For example, such as Figure 7 As shown, Figure 7 (A) represents the initial computation graph, which includes six operators: Slice, Gather, ScatterAdd, MatMul, Mul, and Div. Figure 7In (A), Slice's output is tensor A; Gather's input is tensor A and its output is tensor B; ScatterAdd's input is tensor B and its output is tensor C; MatMul's input is tensor C and its output is tensor D; Mul's input is tensor D and its output is tensor E; Div's input is tensor B and tensor D. If the pre-defined operator fusion rule is: Gather and ScatterAdd are fused, and Mul and Div are fused, then based on this operator fusion rule, the following operations are performed: Figure 7 After operator fusion of (A), we can obtain the following: Figure 7 The calculation diagram shown in (B) is as follows. Figure 7 (B) includes four computational subgraphs, each corresponding to one of four fusion operators: Slice, FuseOp1, MatMul, and FuseOp2. Figure 7 In (B), the output of Slice is tensor A; the input of FuseOp1 is tensor A, and the outputs are tensors B and C; the input of MatMul is tensor C, and the output is tensor D; the input of FuseOp2 is tensor D and tensor B. It is understandable that, due to... Figure 7 (B) is the computation graph obtained after operator fusion, so it can be... Figure 7 Each operator in (B) is called a "fusion operator".

[0119] It is understood that, in the embodiments of this application, the operator fusion step can be selected according to the actual situation. In some embodiments, this step can be performed before the subsequent recalculation step. In other embodiments, this step can be omitted, and the subsequent recalculation step can be performed directly.

[0120] (2) Recalculation

[0121] The recomputation process mainly includes constructing a producer-consumer list, determining a recomputation candidate set, selecting a recomputation point set from the recomputation candidate set, determining the recomputation graph, and generating one or more of the following descriptions.

[0122] A) Construct a producer-consumer list

[0123] Based on the data dependencies between nodes in the computation graph, the nodes are first topologically sorted, for example, according to the physical relationships between the operators in the computation graph. Then, a producer-consumer list is obtained from the sorting result. Here, a producer is a computational subgraph that generates data from two computational subgraphs with data dependencies, i.e., an operator that generates data. A consumer is a computational subgraph that uses data from two computational subgraphs with data dependencies, i.e., an operator that uses data.

[0124] For example, such as Figure 8 As shown, Figure 8 (A) is the computational subgraph obtained after one fusion. Based on the data dependencies between nodes in the computational graph, [the following is done / is performed / etc.]. Figure 8 By performing a topological sort on the nodes in the computation graph shown in (A), we can obtain the following: Figure 8 The sorting result is shown in (B). Figure 8 In (B), Slice and FuseOp1 have a data dependency relationship. FuseOp1 has data dependencies on MatMul and FuseOp2 respectively. MatMul has a data dependency relationship with Div, and Div has a data dependency relationship with FuseOp2. Specifically, for Slice and FuseOp1, Slice can generate data, and FuseOp1 can use the data generated by Slice; therefore, Slice is the producer, and FuseOp1 is the consumer. Similarly, for FuseOp1 and MatMul, FuseOp1 is the producer, and MatMul is the consumer; for FuseOp1 and FuseOp2, FuseOp1 is the producer, and FuseOp2 is the consumer; for MatMul and Div, MatMul is the producer, and Div is the consumer; for Div and FuseOp2, Div is the producer, and FuseOp2 is the consumer. Thus, from... Figure 8 (B) can be used to obtain Figure 8 The producer-consumer list corresponding to the computation graph in (A) can be shown in Table 1.

[0125] Table 1

[0126] Producer Consumer Slice FuseOp1 FuseOp1 MatMul FuseOp1 FuseOp2 MatMul Div Div FuseOp2

[0127] B) Determine the recalculation candidate set

[0128] After constructing the producer-consumer list, for any producer in the list, the set of operators directly and indirectly related to the tensor output by that producer can be used as data in the recomputation candidate set, which can be called the recomputation node set. Thus, the recomputation candidate set is obtained. The recomputation node set can include one node or multiple nodes.

[0129] For example, see further. Figure 8Taking the producer FuseOp1 as an example, the tensors output by this producer are tensor B and tensor C. The node directly related to tensor B is Gather, and there are no nodes indirectly related to tensor B. The node directly related to tensor C is ScatterAdd, and the node indirectly related to tensor C is Gather. Therefore, the recomputation candidate set obtained from the producer FuseOp1 is [(Gather), (Gather, ScatterAdd)]. This recomputation candidate set includes two recomputation node sets: one set is (Gather), which includes one node, namely node Gather, and the other set is (Gather, ScatterAdd), which includes two nodes, namely nodes Gather and ScatterAdd.

[0130] C) Select the set of recomputation nodes from the recomputation candidate set.

[0131] The methods for selecting the set of recomputation points from the recomputation candidate set can include one or more of the following: selection based on memory size, selection based on data mutation, and selection based on operator fusion rules. These three methods can be chosen individually or in any combination, depending on the specific circumstances, and are not limited here. The following describes each of these three methods.

[0132] a) Filtering based on memory size

[0133] When filtering based on memory size, the desired set of recomputation nodes for the desired producer can be selected from the recomputation candidate set based on the memory size occupied by the tensors output by each recomputation node set in the recomputation candidate set. Specifically, as follows... Figure 9As shown, in S901, the memory occupied by the output tensor corresponding to each recomputation node set in the recomputation candidate set is compared with the previously set memory threshold or peak memory threshold. In S902, when the memory occupied by the output tensor corresponding to the recomputation node set is greater than or equal to the previously set memory threshold or peak memory threshold, the recomputation node set is determined to be the required recomputation node set, and it can be retained. In S903, when the memory occupied by the output tensor corresponding to the recomputation node set is less than the previously set memory threshold or peak memory threshold, the recomputation node set is determined not to be the required recomputation node set, and it can be discarded. Thus, the required recomputation node set is selected from the recomputation candidate set. In one example, the output tensor corresponding to the recomputation node set refers to a tensor output by the producer corresponding to the recomputation node set, and each node in the recomputation node set is directly or indirectly related to this tensor. The input tensor corresponding to the set of recomputable nodes refers to at least one tensor of the producer input corresponding to the set of recomputable nodes, and each node in the set of recomputable nodes is directly or indirectly related to each tensor in the at least one tensor.

[0134] As another possible implementation, during the process of obtaining the producer-consumer list, it's possible to record whether an indirect path exists between producers and consumers. That is, to record whether the tensor output by the producer can be indirectly used as the input tensor of the consumer, in addition to being directly used as the input tensor. For example, see [link to relevant documentation]. Figure 8 For the producer-consumer pair FuseOp1-FuseOp2, the tensor B output by producer FuseOp1 can be directly used as the input tensor of consumer FuseOp2, and the tensor C output by producer FuseOp1 can be indirectly used as the input tensor of FuseOp2 via MatMul and Div. Therefore, there is an indirect path between FuseOp1 and FuseOp2.

[0135] For any producer-consumer pair, such as Figure 10As shown, in S1001, it can be determined whether there is an indirect path between the producer and the consumer. When there is no indirect path, the required set of recomputation nodes corresponding to the producer can be selected from the recomputation candidate set based on the memory size occupied by the output tensors of each recomputation node set corresponding to the producer in the recomputation candidate set. Specifically, in S1002, the memory occupied by the output tensors of each recomputation node set corresponding to the producer in the recomputation candidate set can be compared with the previously set memory threshold or peak memory threshold. In S1003, when the memory occupied by the output tensors of the recomputation node set is greater than or equal to the previously set memory threshold, the recomputation node set is determined to be the required set of recomputation nodes, and can be retained. In S1004, when the memory occupied by the output tensors of the recomputation node set is less than the previously set memory threshold or peak memory threshold, the recomputation node set is determined not to be the required set of recomputation nodes, and can be discarded. This allows us to select the desired set of recalculation points from the recalculation candidate set.

[0136] For example, see further. Figure 8 For the producer-consumer pair MatMul-Div, there is no indirect path between them. If the initial memory threshold is set to 40 bytes, the tensor output by the producer MatMul will occupy 50 bytes of memory. Since there is only a direct path between MatMul and Div, and the tensor output by the producer MatMul occupies more memory than the initial memory threshold, MatMul can be used as the set of recomputable nodes required.

[0137] For example, determining the memory occupied by tensors output by a producer can be done, but is not limited to, based on the data size of the tensors output by the producer. Once the computation graph is determined, the structure (e.g., dimension, type) of the tensors output by each operator in the computation graph can be determined. Therefore, the memory size occupied by the tensors output by each operator can be determined based on the structure of the tensors output by each operator. For example, if the tensor has dimensions [4,4,3,2] and a type of float32, since each data of type float32 occupies 4 bytes, the memory size occupied by this tensor is: 4x4x3x2x4 = 384 bytes.

[0138] Continue reading Figure 10In S1005, when an indirect path exists between the producer and consumer, the peak memory usage of each operator on the indirect path during execution can be calculated. Next, in S1006, it is determined whether the peak memory corresponding to at least one operator is greater than or equal to a pre-set peak memory threshold. Then, in S1007, if the peak memory corresponding to at least one operator is greater than or equal to the pre-set peak memory threshold, the set of operators directly and indirectly related to each tensor input from the producer to the consumer in the producer-consumer relationship can be considered as a required set of recomputation nodes. In S1008, if the peak memory corresponding to each operator on the indirect path between the producer and consumer is less than the pre-set peak memory threshold, the set of operators directly and indirectly related to the tensors input from the producer to the consumer in the producer-consumer relationship can be considered as a non-required set of recomputation nodes and discarded. Thus, the required set of recomputation nodes is selected from the recomputation candidate set.

[0139] For example, such as Figure 11 As shown, in Figure 11 FuseOp1 and FuseOp2 can form a producer-consumer relationship. There is an indirect path between FuseOp1 and FuseOp2, with the operator MatMul on this path. If the memory usage of MatMul during execution (i.e., the sum of memory occupied by tensors B, C, D, and E) exceeds the previously set peak memory threshold, then the sets of operators directly and indirectly related to tensor B, and the sets of operators directly and indirectly related to tensor E, can be selected as the required recomputation node sets. Specifically, the set of operators directly and indirectly related to tensor B is Gather, and the set of operators directly and indirectly related to tensor E is (Gather, ScatterAdd, Div). Therefore, the selected recomputation node sets at this point are (Gather) and (Gather, ScatterAdd, Div), respectively.

[0140] In some embodiments, to improve the accuracy of peak memory threshold-based filtering, the size of the memory occupied by the tensor input from the producer to the consumer (i.e., the tensor output by the producer) and the memory occupied by the input tensor related to that tensor required by the producer can be used as an additional judgment condition during filtering. Specifically, when there is an indirect path between the producer and consumer, if the memory occupied by the tensor input from the producer to the consumer (i.e., the tensor output by the producer) is greater than the memory occupied by the input tensor related to that tensor required by the producer, and at least one operator on the indirect path between the producer and consumer has a peak memory value greater than or equal to the previously set peak memory threshold, then the set of operators directly and indirectly related to each tensor input from the producer to the consumer in the producer-consumer path can be considered as a set of required recomputation nodes.

[0141] For example, such as Figure 12 As shown in (A), for the producer-consumer pair of operators 1 and 5, there is an indirect path between them, and the peak memory is highest when operator 4 is executed, at which point the peak memory is (T1+T2+T3+T4). If the peak memory when operator 4 is executed exceeds the previously set peak memory threshold, then operator 1 can be used as the required set of recomputable nodes. Figure 12 As shown in (B), when operator 1 is used as the required set of recomputable nodes, and the peak memory after recomputation is (T0+T2+T3+T4), if T1 is greater than T0, then recomputation can reduce the peak memory of the computation graph. If T1 is less than T0, then recomputation actually increases the peak memory of the computation graph. If T1 equals T0, then recomputation does not change the peak memory of the computation graph, but the system overhead increases due to the recomputation. Therefore, when T1 is greater than T0, operator 1 can be used as the required set of recomputable nodes.

[0142] b) Screening based on data mutation

[0143] When filtering based on data mutations, it can be determined whether a set of recomputation nodes in the candidate recomputation set exhibits a data mutation. If the deviation (e.g., difference, ratio) between the memory size of the output tensor corresponding to the recomputation node set and the memory size of at least one corresponding input tensor is greater than or equal to a pre-set data threshold, it indicates that a data mutation exists, and the recomputation node set is retained; otherwise, it is discarded. This eliminates the possibility of excessive memory usage caused by data mutations.

[0144] For example, see further. Figure 11If the set of recalculation points in the candidate set is [(Gather), (Gather-ScatterAdd-Div)], and the difference between the memory occupied by tensor E and tensor A is greater than the previously set data threshold, then the recalculation point set Gather has a data mutation, and this set can be retained. If the difference between the memory occupied by tensor B and tensor A is less than the previously set data threshold, then the recalculation point set (Gather-ScatterAdd-Div) does not have a data mutation, and this set can be discarded. Therefore, the selected set of recalculation points is (Gather).

[0145] For example, to eliminate excessive memory usage through data mutation, such as... Figure 12 The above, Figure 12 (A) represents the execution order of each operator before recomputation, the lifetime of the input and output tensors of each operator, and the peak memory usage of each operator during execution. Figure 12 (B) represents the execution order of each operator after recomputation, the lifetime of the input and output tensors of each operator, and the peak memory usage of each operator during execution. Figure 12 Comparing (A) and (B), when operator 4 is executed, the peak memory before recalculation is T1+T2+T3+T4, and the peak memory after recalculation is T0+T2+T3+T4. If the value obtained by subtracting T0 from T1 is greater than the previously set data threshold, and operator 1 is one of the recalculation node sets in the determined recalculation candidate set, it can be seen that when operator 4 is executed, the peak memory after recalculation is significantly less than the peak memory before recalculation. Therefore, in this case, the peak memory can be significantly reduced, so operator 1 can be used as the selected recalculation node set. If the value obtained by subtracting T0 from T1 is less than the previously set data threshold, and operator 1 is one of the recalculation node sets in the determined recalculation candidate set, it can be seen that when operator 4 is executed, the peak memory after recalculation is relatively close to the peak memory before recalculation. Therefore, in this case, operator 1 can be used as the selected recalculation node set, or operator 1 can not be used as the selected recalculation node set.

[0146] c) Screening based on operator fusion rules

[0147] When selecting the set of recomputable nodes based on operator fusion rules, such as Figure 13As shown in (A), in S1311, based on pre-defined operator fusion rules, it can be determined whether the set of recomputation nodes in the recomputation candidate set can be fused with the consumer corresponding to that set of recomputation nodes. In S1312, if fusion is possible, the set of recomputation nodes is retained. In S1313, if fusion is not possible, the set of recomputation nodes can be discarded. In one example, S1311 can also be understood as determining whether the nodes in the recomputation node set that output tensors to the outside can be fused with their corresponding consumers.

[0148] As another possible implementation, such as Figure 13 As shown in (B), in S1321, based on pre-defined operator fusion rules, it can be determined whether the set of recomputation nodes in the candidate set of recomputation nodes can be fused with the consumer corresponding to that set of recomputation nodes. In S1322, if fusion is possible, the set of recomputation nodes is retained. In S1323, if fusion is not possible, it can be determined whether there is an indirect path between the producer and consumer corresponding to the set of recomputation nodes. When an indirect path exists, the set of recomputation nodes can be retained, i.e., S1322 is executed. In S1324, when no indirect path exists, the recomputation node combination can be discarded. This is how the set of recomputation nodes is selected. Even when the operator fusion rules are not met, but an indirect path exists, recomputation can still shorten the lifecycle of tensors with excessive memory consumption, thus potentially reducing peak memory usage. In one example, S1321 can also be understood as determining whether the nodes in the set of recomputation nodes that output tensors to the outside can be fused with their corresponding consumers.

[0149] For example, see further. Figure 12 Suppose that the set of recomputation nodes in the determined recomputation candidate set is operator 1, and operator 1 and operator 5 cannot be merged. Since operator 1 and operator 5 do not meet the fusion rules, but there is an indirect path between them, operator 1 can be used as the required set of recomputation nodes. Figure 12 Comparing (A) and (B), if the memory T1 occupied by the tensor output by operator 1 is too large, then... Figure 12 In (B), the lifetime of the tensor output by operator 1 can be significantly shortened, and the peak memory corresponding to the entire computation graph is also significantly reduced.

[0150] D) Determine the recalculation diagram

[0151] After selecting the set of recomputable nodes, a recomputation graph can be determined from this set. For example, edges between the recomputable node set and the consumers corresponding to the producers of that recomputable node set can be used as recomputable nodes in the computation graph. Then, the producer subgraphs corresponding to each recomputable node are copied, with the number of resulting producer subgraphs equal to or greater than the number of selected recomputable node sets. Finally, all nodes in the producer subgraphs except those corresponding to the recomputable node set are deleted, along with edges in the producer subgraphs that are unrelated to the recomputable node set corresponding to that producer subgraph. This yields the recomputation graphs corresponding to each recomputable node set. Since the probability of duplicate recomputable node sets in this way is very small, the probability of duplicate computation is reduced, thus reducing computational cost and saving recomputation overhead. In one example, deleting all nodes in the producer subgraph except those corresponding to the recomputable node set can be understood as deleting operators in the producer subgraph that are unrelated to the tensors output by that recomputable node set.

[0152] In one example, for any set of heavily computed nodes, the producers belonging to that set can be directly copied into the computation graph, thus obtaining the producer subgraph corresponding to that set of heavily computed nodes. Then, by deleting all nodes in the producer subgraph except those belonging to the set of heavily computed nodes corresponding to that producer subgraph, and by deleting edges in the producer subgraph that are unrelated to the set of heavily computed nodes corresponding to that producer subgraph, the recomputation graph corresponding to each set of heavily computed nodes is obtained.

[0153] For example, such as Figure 14 As shown, Figure 14 (A) is the fused subgraph obtained by performing one operator fusion on the initial computation graph. If in Figure 14 In (A), if the memory occupied by tensor B exceeds the user-configured memory threshold, then the set of recomputable nodes can be determined as Gather, and the edge corresponding to tensor B can be determined as a recomputable point. Next, the producer subgraph corresponding to this recomputable point can be copied, i.e., FuseOp1 can be copied, resulting in... Figure 14 The fusion operator shown in (B) is used. Copying the producer subgraph corresponding to the recalculation point can be understood as copying the fusion operator that outputs the tensor B from the fusion subgraph obtained after one operator fusion. Next, in... Figure 14 In (B), operators other than the recomputable node set Gather can be deleted, i.e., ScatterAdd can be deleted, as well as edges unrelated to the recomputable node set Gather can be deleted, i.e., edges between the operator Scatter and the operator Gather, and edges corresponding to the output tensor of the operator Scatter can be deleted, thus obtaining the following... Figure 14 The recalculation diagram is shown in (C).

[0154] E) Generate a new computational graph

[0155] After obtaining the recomputation graph, each recomputation graph is combined with the original computation graph based on the relationships between nodes in each recomputation graph and nodes in the original computation subgraph, thus generating a new computation graph. Specifically, nodes in the recomputation graph that input the first tensor have data dependencies with nodes in the original computation subgraph that output the first tensor, and nodes in the recomputation graph that output the second tensor have data dependencies with nodes in the original computation subgraph that input the second tensor. It can be understood that for the original computation subgraph, if operator fusion was performed on the original computation graph during the recomputation step, then the original computation subgraph is the subgraph obtained after operator fusion; if operator fusion was not performed on the original computation graph during the recomputation step, then the original computation subgraph is the original computation graph.

[0156] For example, directed edges can be constructed between the nodes of the input first target tensor in each recomputation graph and the nodes of the output first target tensor in the original computation graph, and directed edges can be constructed between the nodes of the output second target tensor in each recomputation graph and the nodes of the input second target tensor in the original computation graph, to obtain a new computation graph.

[0157] For example, such as Figure 15 As shown, Figure 15 (A) represents the initial computation graph (i.e., the original computation graph). After performing an operator fusion on this computation graph, we can obtain the following: Figure 15 The first fusion subgraph shown in (B) is an example. Figure 15 After recalculating the first-order fusion subgraph shown in (B), we can obtain the following: Figure 15 The recalculation diagram is shown in (C). Because... Figure 15 The recomputation graph shown in (C) takes tensor A as input and outputs tensor B, while Figure 15 In (B), the operator for output tensor A is operator Slice, and the operator for input tensor B is operator FuseOp2. Therefore, we can... Figure 15 The operators used for input tensor A in the recomputation graph of (C) are... Figure 15 The operators in (B) are connected by Slice, that is, a directed edge is constructed between them, and the Slice is connected by Slice. Figure 15 The operator used to output tensor B in the recomputation graph of (C) and Figure 15 Connect the operators FuseOp2 in (B), that is, construct a directed edge between the two. Therefore, Figure 15 The recalculation graph shown in (C) is similar to... Figure 15 After combining the first fusion subgraphs shown in (B), we can obtain Figure 15 The new computational graph is shown in (D).

[0158] (3) Fusion of quadratic operators

[0159] By performing operator fusion (i.e., secondary fusion) on the newly obtained computation graph, a new fused subgraph can be obtained, thus yielding the optimized computation graph. For example, based on pre-defined operator fusion rules and according to the type of operators in the computation graph (e.g., computationally intensive, memory-intensive, etc.) and the backend characteristics of the hardware, the computation graph can be divided into multiple computation subgraphs. Each computation subgraph can correspond to a fused operator (hereinafter referred to as the "fusion operator"). This results in the optimized computation graph, i.e., the secondary fused subgraph, which is the final optimized computation graph. Then, the final optimized computation graph can be executed. In one example, to execute the final optimized computation graph, the computation graph can be compiled first and then the compiled computation graph can be executed, or the computation graph can be executed directly; regardless of the method, it essentially executes the final optimized computation graph, only the representation is different.

[0160] For example, such as Figure 16 As shown, Figure 16 (A) shows the new computation graph obtained after recalculation. If the pre-defined operator fusion rule is: Gather and Mul fusion, then based on this operator fusion rule... Figure 16 After operator fusion of (A), we can obtain the following: Figure 16 The fusion subgraph shown in (B) is as follows.

[0161] In some embodiments, the node described in this application may refer to an operator, and the operator described in this application may refer to a node, depending on the actual situation.

[0162] Therefore, by first performing operator fusion on the computation graph to obtain a fused subgraph, then identifying the recomputation graph that needs to be recomputed from the fused subgraph, and combining the identified recomputation graph with the fused subgraph to generate a new computation graph, and finally performing operator fusion on the new computation graph to obtain the required computation graph, this method significantly reduces memory usage without introducing large recomputation overhead, thus solving the problem that networks with one or more extremely large tensors cannot be executed.

[0163] To make it easier to understand, the following example illustrates the process of reducing memory usage using the above method.

[0164] For example, such as Figure 17 As shown, in Figure 17 In this context, M and L represent the memory size occupied by the tensor output by the operator, with M representing medium size, such as 56.8M, and L representing very large size, such as 27G. Figure 17The computation graph shown in (A) is the initial computation graph without any processing. Figure 17 In (A), the operators Slice and ScatterAdd share one memory location, and the operators Gather and Div share another memory location. Figure 17 The memory size required for the computation graph shown in (A) is: M1+M2+L1+L2.

[0165] Figure 17 The computational graph shown in (B) is based on a pre-defined operator fusion rule. Figure 17 The computation graph obtained by performing one operator fusion on the computation graph shown in (A) is the first-stage fused subgraph. Figure 17 In (B), the Slice operator and the MatMul operator share the same memory, so Figure 17 The memory size required for the computation graph shown in (B) is: M1+M2+L1+L2.

[0166] Figure 17 The calculation graph shown in (C) is for... Figure 17 The computational graph obtained after recalculating the computational graph shown in (B) is the new computational graph. Figure 17 In (C), operators cannot reuse memory, so Figure 17 The memory size required for the computation graph shown in (C) is: M1+M2+M3+L1+L2.

[0167] Figure 17 The computational graph shown in (D) is based on a pre-defined operator fusion rule. Figure 17 The computation graph obtained by fusing the computation graph shown in (C) using quadratic operators is the quadratic fused subgraph, which is the desired computation graph. Figure 17 In (D), memory cannot be reused between operators, so Figure 17 The memory size required for the computation graph shown in (D) is: M1+M2+M3+L1.

[0168] Assuming M1 = M2 = M3 and L1 = L2, then Figure 17 The computation graph shown in (A) occupies 2M+2L of memory. Figure 17 The computation graph shown in (B) occupies 2M+2L of memory. Figure 17 The computation graph shown in (C) occupies 3M+2L of memory. Figure 17 The computation graph shown in (D) occupies 3M+L memory. Since M represents medium size and L represents very large size, Figure 17 The calculation graph shown in (D) is more complex than... Figure 17 The computation graph of (A) requires less memory (LM). Furthermore, by Figure 17A comparison of (A) and (B) also reveals that the memory size occupied by the computation graph does not change after one operator fusion.

[0169] Furthermore, during the recomputation process, recomputation points can be determined based on memory size, data mutation, and fusion rules. This not only effectively reduces memory usage and improves the efficiency of identifying recomputation points, but also converts memory-intensive tensors into internal storage (such as register space).

[0170] Next, based on the computation graph optimization scheme described above, a computation graph optimization method provided by an embodiment of this application will be introduced. It is understood that this method is another expression of the computation graph optimization scheme described above, and the two are combined. This method is proposed based on the computation graph optimization scheme described above, and some or all of its contents can be found, but are not limited to, the description of the computation graph optimization scheme above.

[0171] Please see Figure 18 , Figure 18 This is a flowchart illustrating a computational graph optimization method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 18 As shown, the computational graph optimization method may include the following steps:

[0172] S1801. Based on the parameters and the data dependencies between nodes in the first computation graph, a second computation graph is obtained from the first computation graph. The parameters include one or more of the following: operator fusion rules, memory threshold of the tensor output by a single node in the first computation graph, peak memory threshold corresponding to the first computation graph, and data mutation threshold corresponding to a node in the first computation graph. The peak memory threshold is the threshold of the memory occupied by all tensors required at the moment a node is executed during the execution of the computation graph. The data mutation threshold is the threshold of at least one node in the first computation graph causing a data mutation.

[0173] Specifically, when optimizing the computation graph, a second computation graph can be obtained from the first computation graph based on parameters and the data dependencies between nodes in the first computation graph. For example, the parameters include one or more of the following: operator fusion rules, a memory threshold for the memory occupied by tensors output by a single node in the first computation graph, a peak memory threshold corresponding to the first computation graph, and a data mutation threshold corresponding to a node in the first computation graph. The peak memory threshold is the threshold for the memory occupied by all tensors required at the moment a node is executed during the computation graph execution process, and the data mutation threshold is the threshold for at least one node in the first computation graph to generate a data mutation. For example, when the computation framework corresponding to the computation graph is the MindSporeAI open-source computation framework, users can set the memory threshold, peak memory threshold, and data mutation threshold through the Context module configured in MindSpore. For example, the memory threshold can also be called a memory threshold, the peak memory threshold can also be called a peak memory threshold, and the data mutation threshold can also be called a data threshold. In some embodiments, a single node can be understood as a single node or as a node obtained by fusing multiple nodes.

[0174] As one possible implementation, when obtaining the second computation graph, N sets of first nodes can be obtained from the producers contained in the first computation graph, based on the parameters and the data dependencies between nodes in the first computation graph. N is a positive integer greater than or equal to 1. Each set of first nodes includes at least one node, wherein each node in a set is directly or indirectly related to a tensor output by a producer in the first computation graph, and is also directly or indirectly related to at least one tensor input by a producer in the first computation graph. Then, each set of first nodes is recomputed to obtain N recomputable subgraphs, which constitute the second computation graph. This yields the second computation graph. For example, the second computation graph may include N recomputable subgraphs. For example, the first node set can be understood as the required recomputable node set described above, and the recomputable subgraph can be understood as the recomputable graph described above.

[0175] In one example, recompiling each set of first nodes to obtain N recomputed subgraphs can be achieved as follows: Specifically, for any set of nodes among the N sets of first nodes, first copy the producer subgraph (also called the producer) corresponding to that set. Then, delete all nodes in the producer subgraph except those contained in any set, and delete edges in the producer subgraph that are unrelated to any set. This gives the recomputed subgraph corresponding to that set. Traversing each of the N sets of first nodes yields N recomputed subgraphs.

[0176] Specifically, to obtain the first set of nodes, a first list can be generated based on the data dependencies between nodes in the first computation graph. This first list includes the correspondence between producers and consumers in the first computation graph. For example, the first list can be the producer-consumer list described above, as shown in Table 1 above. This step can be understood as the step of constructing the producer-consumer list in the recomputation step described above.

[0177] Then, based on the tensors output by each producer in the first list, a candidate node set is obtained. This candidate node set includes at least the first node set. Specifically, for any producer in the first list, the set of nodes directly and indirectly related to the tensors output by that producer is considered as one node set. This step can be understood as the step of determining the recomputation candidate set in the recomputation step described above. The candidate node set can be understood as the recomputation candidate set described above.

[0178] Finally, based on the parameters, a first set of nodes is selected from the candidate node set. This step can be understood as one or more of the steps in the recomputation steps described above, such as filtering based on memory size, filtering based on data mutation, and filtering based on operator fusion rules, to determine the recomputation graph from the recomputation candidate set.

[0179] In one example, the parameter is a memory threshold; the memory occupied by the output tensor corresponding to the first node set exceeds the memory threshold. This automatically selects the node set corresponding to the output tensor that occupies excessive memory as the required node set, thus solving the problem of memory consumption by a single operator and / or tensor.

[0180] In one example, the parameter is the peak memory threshold; there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set exceeds the peak memory threshold. This automatically identifies the node set with excessively high peak memory usage as the required node set, thus solving the memory consumption problem of individual operators and / or tensors.

[0181] In one example, the parameter is the data mutation threshold; the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This automatically selects the node set with data mutations as the required node set, thus eliminating the problem of excessive memory usage caused by data mutations.

[0182] In one example, the parameter is the operator fusion rule; the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer of the first node set. This allows the desired node set to be selected using the operator fusion rule.

[0183] In one example, the parameters are a memory threshold and a peak memory threshold; there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; and / or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold. This allows for the selection of the desired node set by combining the memory threshold and the peak memory threshold, improving the efficiency and accuracy of the selection process.

[0184] In one example, the parameters are a memory threshold and a data mutation threshold; the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, and / or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This allows for the selection of the desired node set by combining the memory threshold and the data mutation threshold, improving the efficiency and accuracy of the selection process.

[0185] In one example, the parameters are a memory threshold and an operator fusion rule; the first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set can be fused with the consumer corresponding to the first node set, or the first node set cannot be fused with the consumer corresponding to the first node set, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows for the selection of the desired node set by combining the memory threshold and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0186] In one example, the parameters are a peak memory threshold and a data mutation threshold; there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; and / or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This allows for the selection of the desired node set by combining the peak memory threshold and the data mutation threshold, improving the efficiency and accuracy of the selection process.

[0187] In one example, the parameters are a memory threshold and an operator fusion rule. The first node set meets one or more of the following conditions: there is an indirect path between the producer and consumer corresponding to the first node set, and the peak memory usage of each node between the producer and consumer of the first node set is higher than the peak memory threshold; or, the first node set and its corresponding consumer can be fused; or, the first node set and its corresponding consumer cannot be fused, and there is an indirect path between the producer and consumer of the first node set. This allows for the selection of the desired node set by combining the peak memory threshold and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0188] In one example, the parameters are a data mutation threshold and an operator fusion rule. The first node set meets one or more of the following conditions: the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set can be fused with the consumer corresponding to the first node set; or, the first node set cannot be fused with the consumer corresponding to the first node set, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows for the selection of the desired node set by combining the data mutation threshold and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0189] In one example, the parameters are a memory threshold, a peak memory threshold, and a data mutation threshold. The first node set meets one or more of the following conditions: there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. This allows for the selection of the desired node set by combining the memory threshold, peak memory threshold, and data mutation threshold, improving the efficiency and accuracy of the selection process.

[0190] In one example, the parameters are a memory threshold, a peak memory threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node during execution between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the first node set and its corresponding consumers can be fused; or, the first node set and its corresponding consumers cannot be fused, and there is an indirect path between the producers and consumers corresponding to the first node set. This allows for the selection of the desired node set by combining the memory threshold, peak memory threshold, and operator fusion rule, improving the efficiency and accuracy of the selection process.

[0191] In one example, the parameters are a memory threshold, a data mutation threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set can be fused with the consumer corresponding to the first node set; or, the first node set cannot be fused with the consumer corresponding to the first node set, and there is an indirect path between the producer and consumer corresponding to the first node set. This allows for the selection of the desired node set by combining the memory threshold, data mutation threshold, and operator fusion rule, improving the efficiency and accuracy of the selection process.

[0192] In one example, the parameters are a peak memory threshold, a data mutation threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set can be fused with its corresponding consumers; or, the first node set cannot be fused with its corresponding consumers, and there is an indirect path between the producers and consumers corresponding to the first node set. This allows for the selection of the desired node set by combining the peak memory threshold, the data mutation threshold, and the operator fusion rule, improving the efficiency and accuracy of the selection process.

[0193] In one example, the parameters are a memory threshold, a peak memory threshold, a data mutation threshold, and an operator fusion rule. The first node set meets one or more of the following conditions: there is no indirect path between the producers and consumers corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; or, there is an indirect path between the producers and consumers corresponding to the first node set, and the peak memory usage of each node during execution between the producers and consumers corresponding to the first node set is higher than the peak memory threshold; or, the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold; or, the first node set and its corresponding consumers can be fused; or, the first node set and its corresponding consumers cannot be fused, and there is an indirect path between the producers and consumers corresponding to the first node set. This allows for the selection of the desired node set by combining the memory threshold, peak memory threshold, data mutation threshold, and operator fusion rule, improving the efficiency and accuracy of the selection process.

[0194] In some embodiments, before S1801, operators in the first computation graph can be fused. This improves the performance of operators in the computation graph; in addition, the number of operators can be reduced through the first operator fusion, thus reducing the overhead of analyzing operators in subsequent recomputation processes; furthermore, it can expand the scope of operator analysis in subsequent recomputation processes, improving the recomputation effect. For example, but not limited to, operators in the first computation graph can be fused based on pre-defined operator fusion rules.

[0195] After obtaining the second computational graph, S1802 can be executed.

[0196] S1802. Merge the second computation graph with the first computation graph to obtain a third computation graph. In the third computation graph, there is a first directed edge between the first node that outputs the first tensor and the second node that inputs the first tensor, and a second directed edge between the third node that outputs the second tensor and the fourth node that inputs the second tensor. The first directed edge points from the first node to the second node, and the second directed edge points from the third node to the fourth node. The first node and the fourth node correspond to nodes in the first computation graph, and the second node and the third node correspond to nodes in the second computation graph.

[0197] Specifically, after obtaining the second computation graph, it can be merged with the first computation graph to obtain the third computation graph. In the third computation graph, there is a first directed edge between the first node outputting the first tensor and the second node inputting the first tensor, and a second directed edge between the third node outputting the second tensor and the fourth node inputting the second tensor. The first directed edge points from the first node to the second node, and the second directed edge points from the third node to the fourth node. The first and fourth nodes correspond to nodes in the first computation graph, and the second and third nodes correspond to nodes in the second computation graph. The process of obtaining the third computation graph can be understood as the step of generating a new computation graph in the recomputation step described above.

[0198] In one example, when the second computation graph consists of N recomputable subgraphs, when merging the second computation graph with the first computation graph, directed edges can be constructed between the nodes of each of the N recomputable subgraphs that input the first target tensor and the nodes of each of the N recomputable subgraphs that output the first target tensor in the first computation graph, and directed edges can also be constructed between the nodes of each of the N recomputable subgraphs that output the second target tensor and the nodes of each of the N recomputable subgraphs that input the second target tensor in the first computation graph, thus obtaining the third computation graph.

[0199] S1803. Merge the operators in the third computation graph to obtain the fourth computation graph.

[0200] Specifically, but not limited to, operators in the third computation graph can be fused based on pre-defined operator fusion rules to obtain the fourth computation graph. For example, the fourth computation graph can also be referred to as the optimized computation graph.

[0201] S1804, Execute the fourth calculation diagram.

[0202] Specifically, after obtaining the fourth computation graph, the computation graph can be executed.

[0203] In one example, to execute the fourth computation graph, the computation graph can be compiled first and then the compiled computation graph can be executed, or the computation graph can be executed directly; regardless of the method, the fourth computation graph is essentially being executed, only the form of representation is different.

[0204] In this way, by recomputing the original computation graph to obtain a recomputable graph, and then merging the recomputable graph with the original computation graph to form a new computation graph, and then performing operator fusion on the new computation graph, the optimized computation graph is obtained. Afterwards, the optimized computation graph can be executed. Thus, by optimizing the computation graph through a combination of recomputation and operator fusion, memory usage is significantly reduced without introducing large recomputation overhead, solving the problem of networks with one or more extremely large tensors being unable to execute.

[0205] Based on the methods described in the above embodiments, this application also provides a chip. Please refer to... Figure 19 , Figure 19 This is a schematic diagram of a chip structure provided in an embodiment of this application. Figure 19 As shown, chip 1900 includes one or more processors 1901 and interface circuitry 1902. Optionally, chip 1900 may also include a bus 1903. Wherein:

[0206] Processor 1901 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 1901 or through software instructions. The processor 1901 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0207] The interface circuit 1902 can be used to send or receive data, instructions or information. The processor 1901 can use the data, instructions or other information received by the interface circuit 1902 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1902.

[0208] Optionally, the chip may also include memory, which may include read-only memory and random access memory, and provide operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0209] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0210] Optionally, the interface circuit 1902 can be used to output the execution results of the processor 1901.

[0211] It should be noted that the functions of the processor 1901 and the interface circuit 1902 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0212] It should be understood that each step of the above method embodiments can be completed by hardware logic circuits or software instructions in a processor.

[0213] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0214] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0215] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0216] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A method of optimizing a computational graph, the method comprising: The method comprises: obtaining a second computation graph from the first computation graph based on parameters and data dependency relationships between nodes in the first computation graph, wherein the parameters comprise one or more of an operator fusion rule, a memory threshold of a tensor output by a single node in the first computation graph, a peak memory threshold corresponding to the first computation graph, and a data mutation threshold corresponding to a node in the first computation graph, wherein the peak memory threshold is a threshold of memory occupied by all tensors that need to be used at a moment when a node is executed in a computation graph execution process, and the data mutation threshold is a threshold at which a data mutation occurs in at least one node in the first computation graph; merging the second computation graph and the first computation graph to obtain a third computation graph, wherein in the third computation graph, a first node outputting a first tensor and a second node inputting the first tensor have a first directed edge, a third node outputting a second tensor and a fourth node inputting the second tensor have a second directed edge, the first directed edge is directed from the first node to the second node, the second directed edge is directed from the third node to the fourth node, the first node and the fourth node correspond to nodes in the first computation graph, and the second node and the third node correspond to nodes in the second computation graph; fusing operators in the third computation graph to obtain a fourth computation graph; executing the fourth computation graph.

2. The method of claim 1, wherein, The method comprises: obtaining N first node sets from producers included in the first computation graph based on the parameters and data dependency relationships between nodes in the first computation graph, wherein N is a positive integer greater than or equal to 1, at least one node is included in each first node set, and the nodes included in one first node set are directly or indirectly related to a tensor output by one producer in the first computation graph and are directly or indirectly related to at least one tensor input by the producer in the first computation graph; recalculating each first node set to obtain N recalculated subgraphs, wherein the N recalculated subgraphs constitute the second computation graph.

3. The method of claim 2, wherein, The parameters are the memory threshold. The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold.

4. The method of claim 2, wherein, The parameters are the peak memory threshold. There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set when the node is executed is higher than the peak memory threshold.

5. The method of claim 2, wherein, The parameters are the data mutation threshold. The deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold.

6. The method of claim 2, wherein, The parameters are the operator fusion rule. The first node set and the consumer corresponding to the first node set can be fused. Or, the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

7. The method of claim 2, wherein, The parameters are the memory threshold and the peak memory threshold; There is no indirect path between the producer and the consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; And / or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node in the producer and the consumer corresponding to the first node set is higher than the peak memory threshold.

8. The method of claim 2, wherein, The parameters are the memory threshold and the data mutation threshold; The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, and / or the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold.

9. The method of claim 2, wherein, The parameters are the memory threshold and the operator fusion rule; The first node set meets one or more of the following conditions: The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

10. The method of claim 2, wherein, The parameters are the peak memory threshold and the data mutation threshold; There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node in the producer and the consumer corresponding to the first node set is higher than the peak memory threshold; And / or, the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold.

11. The method of claim 2, wherein, The parameters are the memory threshold and the operator fusion rule; The first node set meets one or more of the following conditions: There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node in the producer and the consumer corresponding to the first node set is higher than the peak memory threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

12. The method of claim 2, wherein, The parameters are the data mutation threshold and the operator fusion rule; The first node set meets one or more of the following conditions: a deviation value between memory occupied by the output tensor corresponding to the first node set and memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold value, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

13. The method of claim 2, wherein, The parameters are the memory threshold value, the peak memory threshold value and the operator fusion rule. The first node set meets one or more of the following conditions: There is no indirect path between the producer and the consumer corresponding to the first node set, and memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold value, or there is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node in execution is higher than the peak memory threshold value, or a deviation value between memory occupied by the output tensor corresponding to the first node set and memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold value.

14. The method of claim 2, wherein, The parameters are the memory threshold value, the peak memory threshold value and the operator fusion rule. The first node set meets one or more of the following conditions: There is no indirect path between the producer and the consumer corresponding to the first node set, and memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold value, or there is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node in execution is higher than the peak memory threshold value, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

15. The method of claim 2, wherein, The parameters are the memory threshold value, the data mutation threshold value and the operator fusion rule. The first node set meets one or more of the following conditions: Memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold value, or a deviation value between memory occupied by the output tensor corresponding to the first node set and memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold value, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

16. The method of claim 2, wherein, The parameters are the peak memory threshold value, the data mutation threshold value and the operator fusion rule. The first node set meets one or more of the following conditions: The first node set corresponds to a producer and a consumer, and there is an indirect path between the producer and the consumer, and the peak memory of each node in the producer and the consumer is higher than the peak memory threshold, or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

17. The method of claim 2, wherein, The parameters are the memory threshold, the peak memory threshold, the data mutation threshold and the operator fusion rule; The first node set meets one or more of the following conditions: The first node set corresponds to a producer and a consumer, and there is no indirect path between the producer and the consumer, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set corresponds to a producer and a consumer, and there is an indirect path between the producer and the consumer, and the peak memory of each node in the producer and the consumer is higher than the peak memory threshold, or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

18. The method of any one of claims 2-17, wherein, The parameters are the memory threshold, the peak memory threshold, the data mutation threshold and the operator fusion rule; The first node set meets one or more of the following conditions: The first node set corresponds to a producer and a consumer, and there is no indirect path between the producer and the consumer, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set corresponds to a producer and a consumer, and there is an indirect path between the producer and the consumer, and the peak memory of each node in the producer and the consumer is higher than the peak memory threshold, or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set. The parameters are the memory threshold, the peak memory threshold, the data mutation threshold and the operator fusion rule; 19. The method of any one of claims 2-17, wherein, The first node set meets one or more of the following conditions: The first node set corresponds to a producer and a consumer, and there is no indirect path between the producer and the consumer, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set corresponds to a producer and a consumer, and there is an indirect path between the producer and the consumer, and the peak memory of each node in the producer and the consumer is higher than the peak memory threshold, or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set. The parameters are the memory threshold, the peak memory threshold, the data mutation threshold and the operator fusion rule; The first node set meets one or more of the following conditions: The first node set corresponds to a producer and a consumer, and there is no indirect path between the producer and the consumer, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set corresponds to a producer and a consumer, and there is an indirect path between the producer and the consumer, and the peak memory of each node in the producer and the consumer is higher than the peak memory threshold, or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set. The parameters are the memory threshold, the peak memory threshold, the data mutation threshold and the operator fusion rule; The first node set meets one or more of the following conditions: The first node set corresponds to a producer and a consumer, and there is no indirect path between the producer and the consumer, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node set corresponds to a producer and a consumer, and there is an indirect path between the producer and the consumer, and the peak memory of each node in the producer and the consumer is higher than the peak memory threshold, or the deviation between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or the first node set and the consumer corresponding to the first node set can be fused, or the first node set and the consumer corresponding to the first node set cannot be fused, and there is an indirect path between the producer and the consumer corresponding to the first node set.

20. The method of any one of claims 2-17, wherein, The merging the second computation graph with the first computation graph to obtain a third computation graph specifically comprises: respectively constructing a directed edge between a node of an input first target tensor of each of the N re-computation sub-graphs and a node outputting the first target tensor in the first computation graph, and respectively constructing a directed edge between a node of an output second target tensor of each of the N re-computation sub-graphs and a node inputting the second target tensor in the first computation graph, to obtain the third computation graph.

21. The method of any one of claims 1-17, wherein, Before obtaining the second computation graph that needs to be re-computed from the first computation graph based on the parameters and the data dependency relationship between the nodes in the first computation graph, the method further comprises: fusing the operators in the first computation graph.

22. A computational graph optimization apparatus, comprising: comprise: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is configured to execute the method according to any one of claims 1-21.

23. An apparatus, comprising: comprise: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is configured to execute the method according to any one of claims 1-21.

24. A computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is executed on an electronic device, the computer program causes the electronic device to execute the method according to any one of claims 1-21.

25. A computer program product, characterised in that, When the computer program product is executed on an electronic device, the computer program product causes the electronic device to execute the method according to any one of claims 1-21.

26. A chip, characterized by comprise at least one processor and an interface; the interface is configured to provide program instructions or data for the at least one processor; the at least one processor is configured to execute the program instructions to implement the method according to any one of claims 1-21.

Images