Data processing method and device

Abstract

A data processing method and device, the method comprising: acquiring a computational graph of a neural network, the computational graph comprising a plurality of computational nodes and a plurality of directed edges; obtaining a to-be-segmented sub-graph in the calculation graph based on the memory occupied by the tensor data borne by the plurality of directed edges; the to-be-segmented graph comprises at least one computing node in the plurality of computing nodes, and the memory occupied by tensor data borne by directed edges connected with all or part of the computing nodes in the at least one computing node is greater than a first memory threshold; segmenting the to-be-segmented sub-graph into N slice sub-graphs, wherein N is an integer greater than 1; wherein the slice sub-graph comprises at least one computing node, and the input tensor data of the slice sub-graph is a part of the input tensor data of the graph to be segmented; wherein when the calculation graph is executed, the N slice sub-graphs are executed in series, and the tensor calculation result of the to-be-cut sub-graph is obtained based on the execution result of the N slice sub-graphs. The method can reduce the memory overhead of the computational graph.

Description

Technical Field

[0001] The present invention relates to the field of computer technology and specifically to a data processing method and device. Background Art

[0002] Artificial intelligence (AI) is a technical means to simulate human thinking and behavior through computer programs. Artificial intelligence can be defined as a technology that controls relevant sensor devices for perception, uses relevant processors for processing and analysis, acquires knowledge and relies on relevant knowledge for learning and analysis, and finally obtains the best results. At present, artificial intelligence has been applied to image analysis, audio and video processing, natural language processing, recommendation search, scientific computing and other fields. Among them, neural network (NN) is the core component of artificial intelligence. The neural network is based on the brain neuron model and includes many neuron nodes with connection relationships. The neural network uses tensors to perform large-scale high-throughput data calculations to complete complex processing tasks.

[0003] With the continuous development of neural networks, the structure and computational complexity of neural networks continue to rise, and large neural network models have gradually occupied a dominant position. For large neural network models, the demand for computing resources and memory resources is huge. In the actual use cases of large neural network models, there are neural networks involving ultra-large-scale feature calculations. There are extremely large feature maps before and after the neuron nodes of the neural network, resulting in extremely large peak memory. Even the memory occupied by the input and / or output of a single neuron node exceeds the memory capacity of the computing device, which greatly challenges the function of the computing device.

[0004] In view of the above situation, one solution is to use additional registers as a supplement to the memory, which introduces additional hardware resource overhead.

[0005] Another solution is to release the tensor that is not in use temporarily, and recalculate the tensor when it is used later. This solution is only applicable to neural network training scenarios, and it requires repeated calculation of tensors, which increases the cost of computing resources. In particular, this solution cannot be applied when the memory occupied by the input and / or output of a single neuron node exceeds the memory capacity of the computing device. Summary of the invention

[0006] The embodiments of the present application provide a data processing method and device, which can reduce memory overhead and avoid memory overflow while ensuring the execution efficiency of the calculation graph and not increasing hardware resource overhead.

[0007] In a first aspect, a data processing method is provided, the method comprising: obtaining a computational graph of a neural network, the computational graph comprising a plurality of computational nodes and a plurality of directed edges; wherein the two ends of the directed edges are respectively connected to two computational nodes among the plurality of computational nodes; when executing the computational graph, the directed edges carry tensor data, and the computational nodes are used to perform tensor calculations on the tensor data carried by the directed edges connected to the computational nodes; based on the memory occupied by the tensor data carried by the plurality of directed edges, a sub-graph to be segmented is obtained in the computational graph; the graph to be segmented comprises at least one computational node among the plurality of computational nodes, and the memory occupied by the tensor data carried by the directed edges connected to all or part of the computational nodes in at least one computational node is greater than a first memory threshold; the sub-graph to be segmented is segmented into N slice sub-graphs, N being an integer greater than 1; wherein the slice sub-graph comprises at least one computational node, and the input tensor data of the slice sub-graph is part of the input tensor data of the graph to be segmented; wherein, when executing the computational graph, the N slice sub-graphs are executed serially, and the tensor calculation results of the sub-graph to be segmented are obtained based on the execution results of the N slice sub-graphs.

[0008] In this way, the molecular graph to be cut that occupies too much memory in the computation graph can be cut into N slice subgraphs. When executing the computation graph, the N slice subgraphs are executed serially. Among them, compared with the memory required for the execution of the molecular graph to be cut, the memory required for the execution of a single slice subgraph is small, thereby reducing the memory overhead. In addition, the amount of calculation of a single slice subgraph is small, and the amount of calculation of N slice subgraphs is equal to the amount of calculation of the molecular graph to be cut. Therefore, compared with the hardware resources and time required for the execution of the molecular graph to be cut, the hardware resources and time required for the serial execution of the N slice subgraphs have not increased, or have increased very little. In other words, the serial execution of the N slice subgraphs will not affect the execution efficiency of the computation graph and will not increase the overhead of hardware resources.

[0009] In one possible implementation, at a first moment of executing a computational graph, the sum of the memory occupied by tensor data carried by multiple directed edges is greater than a second memory threshold, and the second memory threshold is obtained based on the memory of a computing device that executes the computational graph; based on the memory occupied by the tensor data carried by the multiple directed edges, a sub-graph to be cut is obtained in the computational graph, including: obtaining a directed edge set, the directed edge set including at least one directed edge among multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at a first moment is greater than a first memory threshold, and the first memory threshold is obtained based on the memory occupied by the tensor data carried by the multiple directed edges at the first moment; based on the directed edge set, a sub-graph to be cut is obtained in the computational graph.

[0010] In this implementation, directed edges whose carried tensor data is greater than the memory threshold when memory overflow occurs during the execution of the computation graph are screened, and a subgraph to be sliced ​​is obtained based on the directed edge in the computation graph, so that the subgraph to be sliced ​​includes the directed edge. The sliced ​​subgraph obtained by slicing the subgraph to be sliced ​​including the directed edge can effectively reduce the memory overhead during the execution of the computation graph.

[0011] In one possible implementation, based on the memory occupied by tensor data carried by multiple directed edges, a subgraph to be cut is obtained in the computation graph, including: obtaining a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at a second moment is greater than a first memory threshold, the first memory threshold is preset, and the second moment belongs to the moment of executing the computation graph; based on the directed edge set, the subgraph to be cut is obtained in the computation graph.

[0012] In this implementation, directed edges carrying tensor data greater than a memory threshold are screened, and a subgraph to be sliced ​​is obtained based on the directed edge in the computation graph, so that the subgraph to be sliced ​​includes the directed edge. The sliced ​​subgraph obtained by slicing the subgraph to be sliced ​​including the directed edge can effectively reduce the memory overhead when the computation graph is executed.

[0013] In one possible implementation, a subgraph to be cut is obtained in a computational graph based on directed edges, including: obtaining a first computing node set based on a directed edge set, the first computing node set including computing nodes connected by directed edges in the directed edge set; obtaining tensor data corresponding to computing nodes in the first computing node set, the tensor data corresponding to the computing nodes including output tensor data of the computing nodes and input tensor data of the computing nodes; when the memory occupied by the output tensor data is greater than a third memory threshold, adding the input computing node of the output tensor data to the first computing node set to update the first computing node set; and, when the memory occupied by the input tensor data is greater than the third memory threshold, adding the output computing node of the input tensor data to the first computing node set to update the first computing node set; based on the updated first computing node set, obtaining the subgraph to be cut in the computational graph; wherein the third memory threshold is less than the first memory threshold, and the third memory threshold is obtained based on the first memory threshold.

[0014] Through this implementation, the input nodes and / or output nodes of the tensor data that occupy more memory than the threshold value can be added to the computational node set used to obtain the molecular graph to be sliced. Thus, the molecular graph to be sliced ​​obtained using the computational node set includes the input nodes and / or output nodes of the tensor data that occupy more memory than the threshold value, and the sliced ​​subgraphs obtained by slicing the molecular graph to be sliced ​​can effectively reduce the memory overhead when the computational graph is executed.

[0015] In one possible implementation, based on the updated first computing node set, a subgraph to be cut is obtained in the computing graph, including: obtaining a forward computing node of the updated first computing node set; adding an input node of the forward computing node to the updated first computing node set to obtain a second computing node set; and based on the second computing node set, obtaining the subgraph to be cut in the computing graph.

[0016] In this implementation, the output node of the tensor data used to generate the tensor data occupying more than the threshold memory can be added to the set of computing nodes used to obtain the molecular graph to be sliced. Thus, the molecular graph to be sliced ​​obtained by using the set of computing nodes includes the output node, and the sliced ​​subgraph obtained by slicing the molecular graph to be sliced ​​can effectively reduce the memory overhead when the computation graph is executed.

[0017] In a possible implementation, the size of N is positively correlated with the amount of tensor data carried by the directed edges in the directed edge set.

[0018] In a second aspect, a data processing device is provided, the device comprising: an acquisition module, used to acquire a computational graph of a neural network, the computational graph comprising a plurality of computational nodes and a plurality of directed edges; wherein the two ends of the directed edges are respectively connected to two computational nodes among the plurality of computational nodes; when executing the computational graph, the directed edges carry tensor data, and the computational nodes are used to perform tensor calculations on the tensor data carried by the directed edges connected to the computational nodes; an acquisition module, used to obtain a subgraph to be segmented in the computational graph based on the memory occupied by the tensor data carried by the plurality of directed edges; the graph to be segmented comprises at least one computational node among the plurality of computational nodes, and the memory occupied by the tensor data carried by the directed edges connected to all or part of the computational nodes in at least one computational node is greater than a first memory threshold; a segmentation module, used to segment the subgraph to be segmented into N slice subgraphs, N being an integer greater than 1; wherein the slice subgraph comprises at least one computational node, and the input tensor data of the slice subgraph is part of the input tensor data of the graph to be segmented; wherein, when executing the computational graph, the N slice subgraphs are executed serially, and the tensor calculation results of the subgraph to be segmented are obtained based on the execution results of the N slice subgraphs.

[0019] In one possible implementation, at a first moment of executing a computational graph, the sum of the memory occupied by tensor data carried by multiple directed edges is greater than a second memory threshold, and the second memory threshold is obtained based on the memory of a computing device that executes the computational graph; an obtaining module is used to: obtain a directed edge set, the directed edge set includes at least one directed edge among multiple directed edges, and the memory occupied by tensor data carried by the directed edges in the directed edge set at a first moment is greater than a first memory threshold, and the first memory threshold is obtained based on the memory occupied by tensor data carried by the multiple directed edges at the first moment; based on the directed edge set, a subgraph to be sliced ​​is obtained in the computational graph.

[0020] In one possible implementation, the module is used to: obtain a directed edge set, the directed edge set includes at least one directed edge among multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at a second moment is greater than a first memory threshold, the first memory threshold is preset, and the second moment belongs to the moment of executing the calculation graph; based on the directed edge set, a subgraph to be cut is obtained in the calculation graph.

[0021] In one possible implementation, the module is used to: obtain a first computing node set based on a directed edge set, the first computing node set including computing nodes connected by directed edges in the directed edge set; obtain tensor data corresponding to the computing nodes in the first computing node, the tensor data corresponding to the computing nodes including output tensor data of the computing nodes and input tensor data of the computing nodes; when the memory occupied by the output tensor data is greater than a third memory threshold, add the input computing node of the output tensor data to the first computing node set to update the first computing node set; and, when the memory occupied by the input tensor data is greater than a third memory threshold, add the output computing node of the input tensor data to the first computing node set to update the first computing node set; based on the updated first computing node set, obtain a subgraph to be cut in the computing graph; wherein the third memory threshold is less than the first memory threshold, and the third memory threshold is obtained based on the first memory threshold.

[0022] In one possible implementation, the module is used to: obtain a forward computing node of the updated first computing node set; add an input node of the forward computing node to the updated first computing node set to obtain a second computing node set; and obtain a subgraph to be cut in the computation graph based on the second computing node set.

[0023] In a possible implementation, the size of N is positively correlated with the amount of tensor data carried by the directed edges in the directed edge set.

[0024] In a third aspect, a data processing device is provided, comprising a processor and a memory; the memory is used to store a computer program, the computer program comprising program instructions; the processor is used to execute the program instructions to implement the method provided in the first aspect.

[0025] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0026] In a fifth aspect, an embodiment of the present application further provides a computer program, which includes computer instructions. When the computer instructions are executed by a computer, the computer executes the method provided in the first aspect.

[0027] The beneficial effects of the second to fifth aspects can be referred to the above introduction to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a calculation graph provided in an embodiment of the present application;

[0029] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0030] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0031] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0032] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0033] Figure 6 A flowchart of a data processing method provided in an embodiment of the present application;

[0034] Figure 7 A flowchart of a data processing method provided in an embodiment of the present application;

[0035] Fig. 8A A schematic diagram of a molecular graph to be cut provided in an embodiment of the present application;

[0036] Figure 8B A schematic diagram of a slice sub-graph provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of a computation graph including a slice subgraph provided in an embodiment of the present application;

[0038] Fig.10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0039] Fig.11 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0040] Fig.12 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The scheme provided by the embodiment of the present application will be described below in conjunction with the accompanying drawings. In the embodiment of the present application, "multiple" means two or more, and "multiple" means two or more. "First", "second", etc. are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0042] To facilitate understanding of the solutions provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0043] Computational graph: also known as data flow graph, is a way to describe the computational process of a neural network using a graph structure. The computational graph consists of multiple computational nodes and multiple directed edges. The two ends of the directed edge are respectively connected to two of the multiple computational nodes. The connection relationship between the computational nodes and the directed edges constitutes the topological structure of the computational graph. Therefore, the computational graph belongs to a topological set. The directed edges carry tensor data and indicate the flow direction of the tensor data. A computational node can also be called an operator. A computational node corresponds to a neuron node in a neural network. The tensor data carried by the directed edge pointing to the computational node is the input tensor data of the computational node. The directed edge pointing to the computational node refers to the directed edge whose end point is the computational node. The computational node calculates its input tensor data and generates and outputs one or more tensor data. The directed edge starting from the computational node carries the tensor data output by the computational node.

[0044] in, Figure 1 Schematic diagram of the computation graph 100 is shown. Figure 1As shown, the computation graph 100 includes a plurality of computation nodes, such as computation node 121, computation node 122, computation node 123, and computation node 124, and a plurality of directed edges, such as directed edge 111, directed edge 112, directed edge 113, directed edge 114, directed edge 115, directed edge 116, and directed edge 117. Among them, directed edge 111 points to computation node 121, indicating that tensor data A1 carried by directed edge 111 is input tensor data of computation node 121. The starting points of directed edges 112 and 113 are computation node 121, indicating that tensor data A2 and tensor data A3 output by computation node 121 are carried by directed edge 112 and directed edge 113, respectively. Directed edge 112 and directed edge 113 point to computing node 122, indicating that tensor data A2 carried by directed edge 112 and tensor data A3 carried by directed edge 113 are input tensor data of computing node 122. The starting point of directed edge 114 is computing node 122, indicating that tensor data A4 output by computing node 122 is carried by directed edge 114. Directed edge 114 points to computing node 123, indicating that tensor data A4 carried by directed edge 114 is input tensor data of computing node 123. Directed edge 115 and directed edge 116 point to computing node 124, indicating that tensor data A5 carried by directed edge 115 and tensor data A6 carried by directed edge 116 are input tensor data of computing node 124. The starting point of directed edge 117 is computing node 124, indicating that tensor data A7 output by computing node 124 is carried by directed edge 117.

[0045] The computing nodes may perform one or more computing operations, for example, computing node 121 may perform an expand operation, computing node 122 may perform a MatMul operation, computing node 123 may perform a SoftMax operation, computing node 123 may perform a MatMul operation, and so on.

[0046] Tensor data: It can be referred to as tensor for short. It is an n-dimensional array. Tensor data is the n-dimensional generalization of scalar, 1-dimensional vector and 2-dimensional matrix. The intermediate calculation results in the training and reasoning of neural network models belong to tensors.

[0047] Input node: is a concept for computing nodes. The output of computing node B1 is the input of computing node B2, so computing node B1 is the input node of computing node B2. The output of a computing node usually refers to the tensor data output by the computing node, and the input of computing node B2 usually refers to the tensor data input to the computing node.

[0048] Output node: is a concept for computing nodes. The output of computing node B2 is the input of computing node B1, and computing node B1 is the output node of computing node B2.

[0049] Forward node: usually a concept for a computing node set consisting of at least one computing node. For any computing node B1 in the computing node set, the input nodes of the computing node B1 are analyzed. If the input nodes of the computing node B1 are not all in the forward nodes of the computing node set. In other words, the forward node of the computing node set is the node in the computing node set that receives tensor data input by external nodes. Among them, the external node refers to the computing node outside the computing node set. Among them, the computing node set has a topological structure, that is, there are directed edges between the computing nodes in the computing node set.

[0050] Subgraph: A subgraph of a computational graph refers to all or part of a computational graph, where a subgraph of a computational graph includes some or all of the computational nodes in the computational graph and the directed edges between these computational nodes.

[0051] Slice subgraph: It is obtained by dividing the subgraph, specifically by dividing the tensor data carried by the directed edges in the subgraph. A subgraph can be divided into N slice subgraphs, where N is an integer greater than 1. Among them, the computing nodes in the subgraph are the same as the computing nodes in the slice subgraph of the subgraph, but the tensor data carried by the directed edges in the slice subgraph of the subgraph is a part (for example, one-Nth) of the tensor data carried by the corresponding directed edges in the subgraph.

[0052] Lifecycle: The lifecycle of a computational graph is generally used to characterize the execution order and dependencies of computational nodes in a computational graph. The lifecycle of a computational graph includes the computational node creation phase, the computational node execution phase, and the computational node destruction phase. The computational node creation phase refers to the process in which a computational node is created and added to a computational graph; the computational node execution phase refers to the process in which a computational node performs tensor calculations and outputs calculation results; the computational node destruction phase refers to the process in which a computational node is destroyed after its calculation results are used.

[0053] Control edge: A connection edge between computing nodes or slice subgraphs in a computational graph, used to control the execution order of computing nodes or slice subgraphs.

[0054] An embodiment of the present application provides a data processing method. The method can identify a molecular graph to be cut that occupies too much memory in a computational graph of a neural network, and divide the subgraph into N slice subgraphs. When executing the computational graph, the N slice subgraphs are executed serially. Among them, compared with the memory required for the execution of the molecular graph to be cut, the memory required for the execution of a single slice subgraph is small, thereby reducing memory overhead. In addition, the amount of calculation of a single slice subgraph is small, and the amount of calculation of N slice subgraphs is equal to the amount of calculation of the molecular graph to be cut. Therefore, compared with the hardware resources (such as computing resources, memory resources) and time required for the execution of the molecular graph to be cut, the hardware resources and time required for the serial execution of the N slice subgraphs have not increased, or have increased very little. In other words, the serial execution of the N slice subgraphs does not affect the execution efficiency of the computational graph and does not increase the overhead of hardware resources.

[0055] Next, the data processing method provided in the embodiment of the present application is described in detail.

[0056] First, the method can be executed by any module, device, computing instance, equipment, platform or cluster with data processing function. In the following description, the method is executed by computing device C1 as an example. Figure 2 As shown, the method includes the following steps.

[0057] Step 201, the computing device C1 obtains a computing graph of a neural network, the computing graph includes a plurality of computing nodes and a plurality of directed edges; wherein the two ends of the directed edges are respectively connected to two computing nodes among the plurality of computing nodes; when executing the computing graph, the directed edges carry tensor data, and the computing nodes are used to perform tensor calculations on the tensor data carried by the directed edges connected to the computing nodes. The computing nodes specifically perform tensor calculations on the tensor data carried by the directed edges pointing to the computing nodes.

[0058] The calculation graph may be set to be executed by the calculation device C2, wherein the calculation device C2 and the calculation device C1 may be the same calculation device or different calculation devices.

[0059] In some embodiments, the computation graph may be computation graph 100 .

[0060] In step 202, the computing device C1 obtains a subgraph to be split in the computing graph based on the memory occupied by the tensor data carried by the multiple directed edges; the graph to be split includes at least one computing node among the multiple computing nodes, and the memory occupied by the tensor data carried by the directed edges connecting all or part of the computing nodes in the at least one computing node is greater than a first memory threshold.

[0061] In step 202, the computing device C1 can infer the memory layout at any time during the period when the computing graph is executed by the computing device C2 based on the input data of the computing graph (i.e., the input data of the computing graph, also called the input data of the neural network, which can be the training set or data set of the neural network, and the data is known), the computing operations performed by each computing node in the computing graph, and the topological relationship between each computing node (i.e., data dependency or flow relationship, execution sequence, etc.). The memory layout includes the sum of the memory occupied by the tensor data carried by the multiple directed edges included in the computing graph (i.e., the cumulative sum of the memory occupied by the tensor data carried by each directed edge), and the memory occupied by the tensor data carried by each directed edge. When the sum of the memory occupied by the tensor data carried by the multiple directed edges is too large, or the memory occupied by the tensor data carried by one or more directed edges is too large, a subgraph to be cut is obtained from the computing graph. The details are as follows.

[0062] In some embodiments, Figure 3 As shown, step 202 includes step 2021 and step 2022. In step 2021, during the period when computing device C2 executes the calculation graph, it is determined whether memory overflow occurs in computing device C2. In which, memory overflow can be referred to as generating a peak memory area, which means that the sum of the memory occupied by the multiple directed edges carrying tensor data included in the calculation graph is greater than the second memory threshold. In which, the second memory threshold is obtained based on the memory capacity of computing device C2. That is, the second memory threshold is determined by the memory capacity of computing device C2. In one example, the second memory threshold may be 50%-100% of the memory capacity of computing device C2. In one example, the second memory threshold may be 50% of the memory capacity of computing device C2. In one example, the second memory threshold may be 60% of the memory capacity of computing device C2. In one example, the second memory threshold may be 70% of the memory capacity of computing device C2. In one example, the second memory threshold may be 80% of the memory capacity of computing device C2. In one example, the second memory threshold may be 90% of the memory capacity of computing device C2. In one example, the second memory threshold may be 100% of the memory capacity of the computing device C2.

[0063] It can be set that at time T1 during the period when the computing device C2 executes the computation graph, the computing device C2 has a memory overflow, that is, at time T1, the sum of the memory occupied by the multiple directed edges included in the computation graph and carrying tensor data is greater than the second memory threshold. At this time, step 2022 can be executed, the computing device C1 screens the key directed edges to obtain the directed edge set D1; and based on the directed edge set D1, obtains the subgraph to be cut in the computation graph.

[0064] Wherein, in step 2022, the directed edges whose tensor data carried at time T1 occupy a memory greater than the first memory threshold are screened to obtain key directed edges. That is, the memory occupied by the tensor data carried by the screened key directed edges at time T1 is greater than the first memory threshold. Wherein, the key directed edge is also called the bottleneck directed edge, and the tensor data carried by the key directed edge at time T1 can be called the bottleneck tensor data. Based on the first memory threshold, M key directed edges can be screened out (that is, the memory occupied by the tensor data carried by each of the M key directed edges at time T1 is greater than the first memory threshold), and M is an integer greater than or equal to 1. The M key directed edges constitute the directed edge set D1.

[0065] For the convenience of description, the multiple directed edges included in the computation graph can be referred to as the directed edge set D0. The directed edge set D0 can be expressed as {N1, ..., Np}. The directed edge set D1 can be expressed as {A1, ..., Am}. Among them, the computation graph includes a total of P directed edges, and Nx represents the directed edges in the computation graph. The directed edge set D1 includes a total of M directed edges, and Ax represents the directed edges in the directed edge computation D1. Among them, M is less than or equal to P.

[0066] Exemplarily, the first memory threshold is obtained based on the memory occupied by the tensor data carried by the multiple directed edges included in the computation graph at time T1. Specifically, the memory occupied by the tensor data carried by the M key directed edges at time T1 is greater than the memory occupied by the tensor data carried by other directed edges at time T1. Among them, the other directed edges are directed edges other than the M key directed edges among the multiple directed edges included in the computation graph. The multiple directed edges included in the computation graph can be sorted in order from large to small according to the memory occupied by the tensor data carried at time T1 to obtain a sorting result. The memory occupied by the tensor data carried by the M+1th directed edge in the sorting result at time T1 can be used as the first memory threshold.

[0067] In some embodiments, Figure 4 As shown, step 202 includes step 401 , step 402 , step 403 and step 404 .

[0068] In step 401, the computing device C1 may traverse the directed edges in the computing graph to screen key directed edges. Specifically, the methods of screening key directed edges include the following two methods.

[0069] In the first example of this embodiment, Figure 4 As shown, in step 402, the computing device C1 screens the key directed edges through two screening conditions. The screening condition H1 is that the sum of the memory occupied by the tensor data carried by the multiple directed edges included in the computation graph is greater than the second memory threshold, and the screening condition H2 is that the tensor data carried by the directed edges is greater than the first memory threshold. Figure 3 The description of the illustrated embodiment will not be repeated here.

[0070] In the second example of this embodiment, in step 402, the computing device C1 may screen the key directed edges based only on whether there are directed edges in the multiple directed edges included in the computation graph whose tensor data occupies a memory greater than the first memory threshold. The screened key directed edges constitute the directed edge set D1. Among them, the first memory threshold is preset. Exemplarily, the first memory threshold may be 10%-100% of the memory capacity of the computing device C2 that executes the computation graph. In one example, the first memory threshold may be 10% of the memory capacity of the computing device C2. In one example, the first memory threshold may be 20% of the memory capacity of the computing device C2. In one example, the first memory threshold may be 50% of the memory capacity of the computing device C2. In one example, the first memory threshold may be 80% of the memory capacity of the computing device C2. In one example, the first memory threshold may be 100% of the memory capacity of the computing device C2.

[0071] It can be set that at time T2 during the execution of the computation graph, if the memory occupied by the tensor data carried by at least one of the multiple directed edges included in the computation graph is greater than the first memory threshold, then the at least one directed edge is used as a key directed edge, and the at least one directed edge constitutes a directed edge set D1. That is, in the second example, the computing device C1 can obtain the directed edge set D1, the directed edge set D2 includes at least one of the multiple directed edges included in the computation graph, and the memory occupied by the tensor data carried by the directed edges in the directed edge set D1 at time T2 is greater than the first memory threshold, the first memory threshold is preset, and the time T2 belongs to the time of executing the computation graph.

[0072] In the above manner, a directed edge set D1 may be obtained. Taking the computation graph 100 as an example, the directed edge set D1 may be {directed edge 114, directed edge 115}.

[0073] In step 403, a computing node set can be obtained based on the directed edge set D1. The computing node set is used to obtain a subgraph to be cut in the computation graph. The obtained subgraph to be cut includes the directed edges in the directed edge set D1 and the computing nodes connected by the directed edges.

[0074] In some embodiments, step 403 includes step 4031, and based on the directed edge set D1, a computing node set E1 is obtained, and the computing node set E1 includes computing nodes connected by directed edges in the directed edge set D1. Specifically, in step 4031, the computing nodes connected at both ends of each directed edge in the directed edge set D1, that is, the input node and output node of the directed edge, can be obtained. Among them, the input node of the directed edge is the computing node pointed to by the directed edge, and the output node of the directed edge refers to the computing node that serves as the starting point of the directed edge. The computing nodes connected at both ends of each directed edge in the directed edge set D1 are included in a set to obtain the computing node set E1. The computing node set E1 can be expressed as {AA0,…,AA j}, where AA x Represents a computing node. Taking the computing graph 100 as an example, the directed edge set D1 can be set to be {directed edge 114, directed edge 115}, and the computing node set E1 is {computing node 122, computing node 123, computing node 124}.

[0075] Step 403 also includes step 4032, obtaining the tensor data corresponding to each computing node in the computing node set E1. The tensor data corresponding to the computing node refers to the output tensor data and input tensor data of the computing node. The output tensor data of the computing node refers to the tensor data output by the computing node, and the input tensor data of the computing node refers to the tensor data input to the computing node. Then, according to the memory occupied by the tensor data corresponding to the computing node, the computing node set E1 is updated to obtain the computing node set E2. The details are as follows.

[0076] When the output tensor data of a computing node in the computing node set E1 is not much smaller than the memory occupied by the bottleneck tensor data, if the input node of the output tensor data of the computing node is not in the computing node set E1, the input node of the output tensor data of the computing node is added to the computing node set E1 to update the computing node set E1. The input node of the tensor data refers to the computing node to which the tensor data is to be input, that is, the tensor data is input into the computing node.

[0077] When the input tensor data of a computing node in the computing node set E1 is not much smaller than the memory occupied by the bottleneck tensor data, if the output node of the input tensor data of the computing node is not in the computing node set E1, the output node of the input tensor data of the computing node is added to the computing node set E1 to update the computing node set E1. The output node of the tensor data refers to the computing node that outputs the tensor data.

[0078] Wherein, the third memory threshold may be set based on the above-mentioned first memory threshold. Specifically, the set third memory threshold is less than the first memory threshold. Exemplarily, the third memory threshold may be 5%-50% of the first memory threshold. In one example, the third memory threshold may be 5% of the first memory threshold. In one example, the third memory threshold may be 10% of the first memory threshold. In one example, the third memory threshold may be 30% of the first memory threshold. In one example, the third memory threshold may be 50% of the first memory threshold.

[0079] If the output tensor data of the computing nodes in the computing node set E1 is greater than the third memory threshold, it means that the output tensor data of the computing nodes in the computing node set E1 is not much less than the memory occupied by the bottleneck tensor data. If the input tensor data of the computing nodes in the computing node set E1 is greater than the third memory threshold, it means that the input tensor data of the computing nodes in the computing node set E1 is not much less than the memory occupied by the bottleneck tensor data.

[0080] After the above update, the computing node set E2 can be obtained.

[0081] The tensor data corresponding to the computing nodes in the computing node set E1 can be represented by the tensor data set F1, where the tensor data set F1 is {B 00 , …, B 0m , B 10 , …, B 1n}. Among them, B 0x Represents the input tensor data, B 1x Represents the output tensor data. The computing node set E2 can be represented as {AA0, ..., AA j ,BB 00 ,…,BB 0m ,BB 10 ,…,BB 1n}, where BB 0x Output node representing input tensor data, BB 1x Represents the input node of the output tensor data. Taking the computation graph 100 as an example, the computation node set E1 is {computation node 122, computation node 123, computation node 124}, then the tensor data set F1 is {tensor data A2, tensor data A3, tensor data A4, tensor data A5, tensor data A6, tensor data A7}, and the computation node set E2 is {computation node 122, computation node 123, computation node 124}.

[0082] In one example, if Figure 5As shown, step 4032 includes step 501, step 502, step 503, step 504, step 505, step 506, and step 507. In step 501, the computing device C1 obtains the computing node set E1. In step 502, the computing device C1 obtains the output tensor data of the computing nodes in the computing node set E1. In step 503, it is determined whether the output tensor data is much smaller than the bottleneck tensor data. If the judgment result of step 503 is no, then in step 504, the input node of the output tensor data is added to the computing node set E1. If the judgment result of step 503 is yes, then in step 505, the input tensor data of the computing nodes in the computing node set E1 is obtained. In step 506, it is determined whether the input tensor data is much smaller than the bottleneck tensor data. If the judgment result of step 506 is no, then in step 507, the output node of the input tensor data is added to the computing node set E1. If the judgment result of step 506 is yes, the updating of the computing node set E1 is terminated to obtain the computing node set E2.

[0083] The obtained computing node set E2 is used to obtain a sub-graph to be cut in the computing graph, wherein the obtained sub-graph to be cut includes computing nodes in the computing node set E2 and directed edges between computing nodes.

[0084] In some embodiments, continue to refer to Figure 4 , step 403 includes step 4033, updating the computing node set E2 according to the relevance of the nodes to obtain the computing node set E3. Specifically, in step 4033, the forward node of the computing node set E2 can be obtained. The forward node of the computing node set E2 is a computing node in the computing node set E2, specifically a computing node in the computing node set E2 that receives tensor data input by external nodes. The external node here refers to a computing node outside the computing node set E2. The input node of the forward node of the computing node set E2 can be added to the computing node set E2 to update the computing node set E2. The updated computing node set E2 can be called the computing node set E3.

[0085] In one example of this embodiment, Figure 6 As shown, step 4033 includes step 601, step 602, step 603, step 604, and step 605. In step 601, a computing node set E2 is obtained. In step 602, a forward node set G1 of the computing node set E2 is obtained. The computing node set E2 may have at least one forward node, and the at least one forward node constitutes the forward node set G1. The forward node set G1 can be represented as {C0, ..., C m}, Cx represents the forward node of the computation node set E2.

[0086] After executing step 602, step 603 may be executed to determine whether the input nodes of at least two forward nodes in the forward node set G1 are the same computing node, and the computing node has no other output nodes. Specifically, in step 603, the forward node set G1 is traversed, and if the input nodes of at least two forward nodes in the forward node set G1 are the same computing node, and the computing node has no other output nodes except the at least two forward nodes, step 604 is executed to add the computing node to the forward node set G1 to update the forward node set G1.

[0087] After step 604, step 602 may be executed again. The updated forward node set G1 is obtained by executing step 602 again. Then, in step 603, it is determined that the input nodes of at least two forward nodes in the updated forward node set G1 are the same computing node, and the computing node has no other output nodes. And so on.

[0088] Until the judgment result of step 603 is no, that is, there are no new at least two forward nodes in the forward node set G1 whose input nodes are the same computing node, or there are new at least two forward nodes in the forward node set G1 whose input nodes are the same computing node, but the computing node has additional output nodes. At this time, step 605 is executed to add the computing nodes in the updated forward node G1 to the computing node set E2 to update the computing node set E2 to the computing node set E3. Specifically, the computing nodes that the updated forward node G1 has but the computing node set E2 does not have are added to the computing node set E2 to obtain the computing node set E3.

[0089] In one example, taking the computation graph 100 as an example, the computation node set E2 is {computing node 122, computing node 123, computing node 124}. After step 4033, the obtained computation node set E3 is {computing node 121, computing node 122, computing node 123, computing node 124}, that is, after step 4033, the computing node 121 is added.

[0090] The computing node set E3 is obtained to obtain the subgraph to be cut in the computing graph. The obtained subgraph to be cut includes the computing nodes in the computing node set E3 and the directed edges between the computing nodes. Specifically, the computing nodes in the computing node set E3 and the directed edges between the computing nodes constitute the subgraph to be cut.

[0091] Continue reading Figure 2After obtaining the molecular graph to be sliced, the computing device C1 executes step 203 to slice the molecular graph to be sliced ​​into N slice subgraphs, where N is an integer greater than 1; wherein the slice subgraph includes the at least one computing node, and the input tensor data of the slice subgraph is part of the input tensor data of the molecular graph to be sliced. When executing the computing graph, the N slice subgraphs are executed serially, and the tensor calculation result of the molecular graph to be sliced ​​is obtained based on the execution results of the N slice subgraphs.

[0092] The computing nodes in the slice subgraph are the same as the computing nodes in the subgraph to be sliced. The slice subgraph and the subgraph to be sliced ​​also include at least one directed edge between the computing nodes. The at least one computing node here can be a computing node connected by a directed edge in the edge set D1, or a computing node set E2, or a computing node set E3.

[0093] The input tensor data of a subgraph refers to the tensor data input to the forward nodes of the subgraph, that is, the tensor data input to the subgraph from the computational nodes outside the subgraph. The input tensor data of a sliced ​​subgraph is part of the input tensor data of the subgraph to be sliced, and accordingly, the tensor data carried by the directed edges in the sliced ​​subgraph is part of the tensor data carried by the corresponding directed edges in the subgraph to be sliced.

[0094] In some embodiments, the N slice subgraphs may be obtained by slicing the input tensor data of the subgraph to be sliced. The input tensor data of the subgraph to be sliced ​​may be sliced ​​into N parts, and one part of the tensor data is input into one slice subgraph. In one example, the N parts of the tensor data may be N equal parts of the tensor data. In one example, the data amounts of different tensor data in the N parts of the tensor data may be different.

[0095] In some embodiments, a split operator is added between the forward nodes of the N slicing subgraphs and the input nodes of the forward nodes, and the split operator is used to split the input tensor data of the subgraph to be split into N parts, and input one part of the tensor data into one slicing subgraph. The split operator is a computing node for splitting the input tensor data of the subgraph to be split.

[0096] In some embodiments, see Figure 7 In step 701, the computing device C1 obtains the subgraph to be segmented. In step 702, the computing device C1 determines a segmentation strategy for the directed edges in the subgraph to be segmented according to the principle of giving priority to the associated directed edges (ie, the directed edges in the directed edge set D1).

[0097] In one example, the size of N is positively correlated with the amount of tensor data carried by the directed edges in the directed edge set D1. That is, in step 702, the segmentation strategy determined is: based on the amount of tensor data carried by the directed edges in the directed edge set D1, the number of slice subgraphs to be segmented, that is, the size of N, is determined.

[0098] In one example, the splitting strategy is to split based on the dimension of the tensor data carried by the directed edge in the edge set D1. The tensor data carried by the directed edge in the edge set D1 can be set to Q dimensions of data, where Q is an integer greater than 1. The splitting strategy can be to split in any one or more of the Q dimensions.

[0099] The computing device C1 can determine in step 703 whether the determined segmentation strategy is suitable for all directed edges in the sub-graph to be segmented. If it is not suitable (for example, the dimensions of the tensor data carried by different directed edges are different, and the dimension adopted by the segmentation strategy is not possessed by the tensor data carried by each directed edge), then execute step 704 to split the sub-graph to be segmented into multiple secondary sub-graphs to be segmented. Then, the segmentation strategy is determined based on the granularity of the secondary sub-graph to be segmented, and segmentation is performed. Among them, the computing nodes in each of the multiple secondary sub-graphs to be segmented are connected, that is, connected by directed edges to ensure that the segmentation method is continuous.

[0100] After determining the segmentation strategy, the computing device C1 may execute step 705 to segment the subgraph to be segmented (or the secondary subgraph to be segmented) based on the segmentation strategy to obtain N slice subgraphs. Then, step 706 is executed to update the computation graph.

[0101] Wherein, step 706 includes step 7061, inserting a segmentation operator between the input nodes of the N slice subgraphs and the molecular graph to be segmented, and inserting a merge (concat) operator between the slice subgraph and the output node of the molecular graph to be segmented. Wherein, the input node of the molecular graph to be segmented refers to the computing node located upstream of the molecular graph to be segmented in the data flow direction of the computational graph, specifically refers to the input node of the forward node of the molecular graph to be segmented, and the input node is a computing node outside the molecular graph to be segmented. The output node of the molecular graph to be segmented refers to the computing node located downstream of the molecular graph to be segmented in the data flow direction of the computational graph. The output node of the molecular graph to be segmented is the output node of the computing node in the molecular graph to be segmented, and the output node of the molecular graph to be segmented is a computing node outside the molecular graph to be segmented. The merge operator is used to merge the execution results of the N slice subgraphs (i.e., the tensor data output by the N slice subgraphs) to obtain the execution result of the molecular graph to be segmented.

[0102] Step 706 includes step 7062, inserting control edges between adjacent slice subgraphs to ensure that the N slice subgraphs are executed serially. The N slice subgraphs may be arranged sequentially in the computation graph, and control edges may be inserted between two adjacent slice subgraphs, the control edges being used to indicate that the execution time of the latter of the two slice subgraphs is after the execution time of the former slice subgraph, to ensure that the N slice subgraphs are executed serially.

[0103] In some embodiments, taking the calculation graph 100 as an example, through steps 201 and 202, the following can be obtained: Fig. 8A After step 203, N molecules are obtained. Figure 8B The sliced ​​subgraph is shown. The tensor data carried by the directed edges in the subgraph to be sliced ​​is a three-dimensional tensor data. In step 203, the slice is performed on the first dimension (the first dimension can also be called the first axis). N is 256, that is, the tensor data is sliced ​​into 256 parts, and the subgraph to be sliced ​​is sliced ​​into slice subgraphs. Figure 1 , ..., slice subgraph 256 and other 256 slice subgraphs. Among them, tensor data A2 is divided into 256 tensor data such as tensor data A2-1, tensor data A3 is divided into 256 tensor data such as tensor data A3-1, tensor data A4 is divided into 256 tensor data such as tensor data A4-1, and tensor data A5 is divided into 256 tensor data such as tensor data A5-1.

[0104] The updated computation graph 100 is as follows Fig. 9 As shown. The directed edge 111 originally pointing to the computing node 121 points to the splitting operator I1. The splitting operator I1 splits the tensor data A1 carried by the directed edge 111 into tensor data A1-1, ..., tensor data A1-256, and inputs the tensor data A1-1, ..., tensor data A1-256 to the splitting slices through the directed edges 111-1, ..., directed edges 111-256. Figure 1 , ..., slice subgraph 256. The directed edge 116 originally pointing to the computing node 124 points to the slicing operator I2. The slicing operator I2 slicing the tensor data A6 carried by the directed edge 116 into tensor data A6-1, ..., tensor data A6-256, and inputs the tensor data A6-1, ..., tensor data A6-256 into the slice subgraph through the directed edges 116-1, ..., directed edges 116-256. Figure 1 ,…, slice subgraph 256.

[0105] Slice subgraphs are executed serially, where there is a control edge between adjacent slice subgraphs, and the control edge controls the execution of the two slice subgraphs connected to it. Figure 1 The execution result of is tensor data A7-1, ..., and the execution result of slice subgraph 256 is tensor data A7-256. Figure 1 The tensor data A7-1 is input to the merge operator through the directed edge 117-1, ..., and the slice subgraph 256 inputs the tensor data A7-256 to the merge operator through the directed edge 117-256. The merge operator merges the execution results of each slice subgraph to obtain the tensor data A7, and inputs the tensor data A7 to the next computing node through the directed edge 117.

[0106] In summary, the data processing method provided by the embodiment of the present application automatically identifies the memory optimization area, effectively identifies the area that needs to be optimized in the overall calculation graph, and avoids invalid optimization. The method can automatically traverse and update the subgraph area to make memory optimization more efficient, avoid duplication, and avoid negative optimization. The method can automatically expand the scope of the subgraph to be sliced, and reduce the repeated input and output tensor data caused by slicing. The method can automatically divide the subgraph that occupies too much memory into multiple copies, and perform serial calculations without introducing additional processing equipment to achieve memory optimization. The method can also introduce control flow to ensure that the life cycles between sliced ​​subgraphs are staggered, avoiding excessive memory overhead caused by parallel computing.

[0107] The present application embodiment provides a data processing device 1000. The data processing device 1000 may also be referred to as a computing device. Fig.10 As shown, the data processing device 1000 includes:

[0108] The acquisition module 1010 is used to acquire a computation graph of a neural network, wherein the computation graph includes a plurality of computation nodes and a plurality of directed edges; wherein the two ends of the directed edge are respectively connected to two computation nodes among the plurality of computation nodes; when executing the computation graph, the directed edge carries tensor data, and the computation node is used to perform tensor calculation on the tensor data carried by the directed edge connected to the computation node;

[0109] Obtaining module 1020, for obtaining a subgraph to be segmented in the computational graph based on the memory occupied by the tensor data carried by the multiple directed edges; the graph to be segmented includes at least one computing node among the multiple computing nodes, and the memory occupied by the tensor data carried by the directed edges connected to all or part of the computing nodes in the at least one computing node is greater than a first memory threshold;

[0110] A segmentation module 1030 is used to segment the subgraph to be segmented into N slice subgraphs, where N is an integer greater than 1; wherein the slice subgraph includes the at least one computing node, and the input tensor data of the slice subgraph is part of the input tensor data of the graph to be segmented;

[0111] Wherein, when executing the computation graph, the N slice subgraphs are executed serially, and based on the execution results of the N slice subgraphs, the tensor calculation results of the subgraphs to be sliced ​​are obtained.

[0112] In some embodiments, at a first moment of executing the computational graph, the sum of the memory occupied by the tensor data carried by the multiple directed edges is greater than a second memory threshold, and the second memory threshold is obtained based on the memory of a computing device that executes the computational graph; the obtaining module 1020 is used to: obtain a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at the first moment is greater than the first memory threshold, and the first memory threshold is obtained based on the memory occupied by the tensor data carried by the multiple directed edges at the first moment; based on the directed edge set, the subgraph to be cut is obtained in the computational graph.

[0113] In some embodiments, the obtaining module 1020 is used to: obtain a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at the second moment is greater than the first memory threshold, the first memory threshold is preset, and the second moment belongs to the moment of executing the computational graph; based on the directed edge set, the subgraph to be cut is obtained in the computational graph.

[0114] In some embodiments, the obtaining module 1020 is used to: obtain a first computing node set based on the directed edge set, the first computing node set including computing nodes connected by directed edges in the directed edge set; obtain tensor data corresponding to computing nodes in the first computing node, the tensor data corresponding to the computing nodes including output tensor data of the computing nodes and input tensor data of the computing nodes; when the memory occupied by the output tensor data is greater than a third memory threshold, add the input computing node of the output tensor data to the first computing node set to update the first computing node set; and, when the memory occupied by the input tensor data is greater than the third memory threshold, add the output computing node of the input tensor data to the first computing node set to update the first computing node set; based on the updated first computing node set, obtain the subgraph to be cut in the computational graph; wherein the third memory threshold is less than the first memory threshold, and the third memory threshold is obtained based on the first memory threshold.

[0115] In an example of this embodiment, the obtaining module 1020 is used to: obtain the forward computing node of the updated first computing node set; add the input node of the forward computing node to the updated first computing node set to obtain the second computing node set; and obtain the subgraph to be cut in the computation graph based on the second computing node set.

[0116] In some embodiments, the size of N is positively correlated with the amount of tensor data carried by the directed edges in the directed edge set.

[0117] The functions of the functional modules in the data processing device 1000 can refer to the above descriptions of the various method embodiments (eg Figure 2 The method embodiment shown is introduced and implemented.

[0118] The present application embodiment provides a data processing device 1100. The data processing device 1100 may also be referred to as a computing device. Fig.11 As shown, the data processing device 1100 includes a processor 1110 and a memory 1120; the memory 1120 is used to store a computer program, and the computer program includes program instructions; when the program instructions are executed by the processor 1110, the data processing device 1100 executes the data processing method described above, for example Figure 2 The method shown.

[0119] In some embodiments, the data processing device 1100 may be implemented as a server or a virtual computing instance (eg, a virtual machine, a container, etc.).

[0120] In some embodiments, the processor 1110 and the memory 1120 may be deployed in the same data processing device 1100 .

[0121] In some embodiments, the data processing device 1100 may be implemented as a computing device cluster. Fig.12 As shown, the data processing device 1100 may include a computing device 1100A and a computing device 1100B. The computing device 1100A and the computing device 1100B are connected via a network. The processor 1110 may be deployed in the computing device 1100A, and the memory 1120 may be deployed in the computing device 1100B.

[0122] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is executed by a processor, the data processing method described above is implemented, for example Figure 2 The method shown.

[0123] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by the processor or a device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes a computer program that instructs the processor to execute the data processing method described above, such as Figure 2 The method shown.

[0124] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain a computational graph of a neural network, the computational graph comprising a plurality of computational nodes and a plurality of directed edges; wherein two ends of the directed edge are respectively connected to two computational nodes among the plurality of computational nodes; when executing the computational graph, the directed edge carries tensor data, and the computational node is used to perform tensor calculations on the tensor data carried by the directed edge connected to the computational node; Based on the memory occupied by the tensor data carried by the multiple directed edges, a subgraph to be split is obtained in the computational graph; the subgraph to be split includes at least one computing node among the multiple computing nodes, and the memory occupied by the tensor data carried by the directed edges connected to all or part of the computing nodes in the at least one computing node is greater than a first memory threshold; Divide the subgraph to be divided into N slice subgraphs, where N is an integer greater than 1; wherein the slice subgraph includes the at least one computing node, and the input tensor data of the slice subgraph is part of the input tensor data of the graph to be divided; Wherein, when executing the computation graph, the N slice subgraphs are executed serially, and based on the execution results of the N slice subgraphs, the tensor calculation results of the subgraphs to be sliced ​​are obtained.

2. The method according to claim 1, characterized in that At a first moment of executing the computation graph, a sum of memory occupied by tensor data carried by the plurality of directed edges is greater than a second memory threshold, where the second memory threshold is obtained based on a memory of a computing device executing the computation graph; The method of obtaining a subgraph to be segmented in the computation graph based on the memory occupied by the tensor data carried by the plurality of directed edges includes: Obtain a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and a memory occupied by tensor data carried by the directed edges in the directed edge set at the first moment is greater than the first memory threshold, and the first memory threshold is obtained based on the memory occupied by the tensor data carried by the multiple directed edges at the first moment; Based on the directed edge set, the subgraph to be cut is obtained in the computation graph.

3. The method according to claim 1, characterized in that The method of obtaining a subgraph to be segmented in the computation graph based on the memory occupied by the tensor data carried by the plurality of directed edges includes: Obtain a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at a second moment is greater than the first memory threshold, the first memory threshold is preset, and the second moment belongs to the moment of executing the computation graph; Based on the directed edge set, the subgraph to be cut is obtained in the computation graph.

4. The method according to claim 2 or 3, characterized in that: The obtaining the subgraph to be cut in the computation graph based on the directed edge includes: Based on the directed edge set, a first computing node set is obtained, wherein the first computing node set includes computing nodes connected by directed edges in the directed edge set; Acquire tensor data corresponding to a computing node in the first computing node, where the tensor data corresponding to the computing node includes output tensor data of the computing node and input tensor data of the computing node; When the memory occupied by the output tensor data is greater than a third memory threshold, adding the input computing node of the output tensor data to the first computing node set to update the first computing node set; and, When the memory occupied by the input tensor data is greater than the third memory threshold, adding the output computing node of the input tensor data to the first computing node set to update the first computing node set; Based on the updated first computing node set, the subgraph to be cut is obtained in the computing graph; The third memory threshold is smaller than the first memory threshold, and the third memory threshold is obtained based on the first memory threshold.

5. The method according to claim 4, characterized in that The obtaining the subgraph to be segmented in the computation graph based on the updated first computing node set includes: Obtain the updated forward computing node of the first computing node set; Adding the input node of the forward computing node to the updated first computing node set to obtain a second computing node set; Based on the second computing node set, the subgraph to be segmented is obtained in the computing graph.

6. The method according to any one of claims 2 to 5, characterized in that: The size of N is positively correlated with the amount of tensor data carried by the directed edges in the directed edge set.

7. A data processing device, characterized in that: The device comprises: The two ends of the directed edge are respectively connected to two computing nodes among the multiple computing nodes; when executing the computing graph, the directed edge carries tensor data, and the computing node is used to perform tensor calculation on the tensor data carried by the directed edge connected to the computing node; An obtaining module is used to obtain a subgraph to be segmented in the computational graph based on the memory occupied by the tensor data carried by the multiple directed edges; the graph to be segmented includes at least one computing node among the multiple computing nodes, and the memory occupied by the tensor data carried by the directed edges connected to all or part of the computing nodes in the at least one computing node is greater than a first memory threshold; A segmentation module, used for segmenting the subgraph to be segmented into N slice subgraphs, where N is an integer greater than 1; wherein the slice subgraph includes the at least one computing node, and the input tensor data of the slice subgraph is part of the input tensor data of the graph to be segmented; Wherein, when executing the computation graph, the N slice subgraphs are executed serially, and based on the execution results of the N slice subgraphs, the tensor calculation results of the subgraphs to be sliced ​​are obtained.

8. The device according to claim 7, characterized in that At a first moment of executing the computation graph, a sum of memory occupied by tensor data carried by the plurality of directed edges is greater than a second memory threshold, where the second memory threshold is obtained based on a memory of a computing device executing the computation graph; The module is used to: Obtain a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and a memory occupied by tensor data carried by the directed edges in the directed edge set at the first moment is greater than the first memory threshold, and the first memory threshold is obtained based on the memory occupied by the tensor data carried by the multiple directed edges at the first moment; Based on the directed edge set, the subgraph to be cut is obtained in the computation graph.

9. The device according to claim 7, characterized in that The module is used to: Obtain a directed edge set, the directed edge set including at least one directed edge among the multiple directed edges, and the memory occupied by the tensor data carried by the directed edges in the directed edge set at a second moment is greater than the first memory threshold, the first memory threshold is preset, and the second moment belongs to the moment of executing the computation graph; Based on the directed edge set, the subgraph to be cut is obtained in the computation graph.

10. The device according to claim 8 or 9, characterized in that The module is used to: Based on the directed edge set, a first computing node set is obtained, wherein the first computing node set includes computing nodes connected by directed edges in the directed edge set; Acquire tensor data corresponding to a computing node in the first computing node, where the tensor data corresponding to the computing node includes output tensor data of the computing node and input tensor data of the computing node; When the memory occupied by the output tensor data is greater than a third memory threshold, adding the input computing node of the output tensor data to the first computing node set to update the first computing node set; as well as, When the memory occupied by the input tensor data is greater than the third memory threshold, adding the output computing node of the input tensor data to the first computing node set to update the first computing node set; Based on the updated first computing node set, the subgraph to be cut is obtained in the computing graph; The third memory threshold is smaller than the first memory threshold, and the third memory threshold is obtained based on the first memory threshold.

11. The device according to claim 10, characterized in that The module is used to: Obtain the updated forward computing node of the first computing node set; Adding the input node of the forward computing node to the updated first computing node set to obtain a second computing node set; Based on the second computing node set, the subgraph to be segmented is obtained in the computing graph.

12. The device according to any one of claims 8 to 11, characterized in that The size of N is positively correlated with the amount of tensor data carried by the directed edges in the directed edge set.

13. A data processing device, characterized in that: The system comprises a processor and a memory; the memory is used to store a computer program, and the computer program comprises program instructions; the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 6.

Images