Self-adaptive re-calculation and load division method, device and equipment based on neural network, and computer readable medium

Abstract

The invention provides a parallel training method, device and equipment based on adaptive re-calculation and load division and a computer readable medium. The method comprises the following steps: firstly, converting a dynamic calculation graph in a deep learning framework into a static calculation graph data structure; then, the calculation sequence is divided, and calculation sub-graphs are generated; constructing a mixed integer programming model according to the calculation time of the operator, the transfer cost of the operator and the constraint condition; then, solving the scheduling optimization problem of the operator by using a solver to generate a video memory optimization strategy; aiming at the generated video memory optimization strategy, evaluating the execution time of each calculation sub-graph, and calculating to obtain a load balancing rate; and finally, according to a calculation result of the load balancing rate, determining a calculation sub-graph which is finally divided, and performing parallel training by utilizing the calculation sub-graph which is finally divided. Through the method provided by the invention, the final operator execution sequence can be found, so that the peak video memory consumption during the operation of the computational graph is within a given limit, and the training time of the neural network model is minimized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for adaptive recalculation and load division based on a neural network. Background Art

[0002] In recent years, benefiting from the improvement of computer software and hardware technology, artificial intelligence (AI) technology, especially deep learning (DL) technology, has developed rapidly and achieved amazing results in many fields. As the core technology of deep learning, the development of deep neural network (DNN) plays a vital role in improving the capabilities of modern artificial intelligence technology.

[0003] However, the current cutting-edge deep neural network algorithms usually use a large amount of training data combined with a complex hierarchical model structure to further improve the comprehensive performance of deep neural networks in various application fields, so that a single device is difficult to meet the computing and storage requirements of deep neural networks. When training large models, the industry usually needs to adopt a carefully designed neural network parallel training strategy, but an efficient parallel strategy requires a lot of time and effort to design and debug based on expert experience. Therefore, how to adaptively adopt a suitable parallel training strategy for a specific network on a specified hardware topology has also become a research hotspot in academia and industry in recent years. Summary of the invention

[0004] The present invention describes a parallel training method, apparatus, device and computer-readable medium based on adaptive recalculation and load partitioning, which can solve the above technical problems.

[0005] According to a first aspect, a parallel training method based on adaptive recalculation and load partitioning is provided, the method comprising:

[0006] Convert the dynamic computation graph in the deep learning framework into a static computation graph; specifically, the static computation graph has a fixed data structure;

[0007] The operator set in the computation graph is topologically sorted according to the dependency relationship to construct a computation sequence, and the computation sequence is divided to generate a computation subgraph; the operator is the basic block for constructing the computation graph and is used to perform computation tasks;

[0008] A mixed integer programming model is constructed based on the operator's computation time, the operator's transfer cost, and the constraints. Specifically, the operator's computation time in the mixed integer programming model reflects the computational performance of parallel training and video memory optimization, and the operator's transfer cost reflects the communication performance of the hardware.

[0009] Use the solver to solve the scheduling optimization problem of operators in the mixed integer programming model to generate a video memory optimization strategy;

[0010] Based on the generated video memory optimization strategy, the execution time of each computational subgraph is evaluated and the load balancing rate is calculated.

[0011] The final divided computational subgraph is determined according to the calculation result of the load balancing rate; and the final divided computational subgraph is used for parallel training.

[0012] In some embodiments, after the calculation sequence is divided, the calculation sequence is further fine-tuned, and the fine-tuning method specifically includes:

[0013] For each operator in the calculation sequence, check whether the partition results of its successor and predecessor nodes are the same as those of the current node;

[0014] If the partition results are different, try to move the operator and the operators before or after the topological sorting to the subsequence where the corresponding partition is located to determine whether the communication volume can be reduced while maintaining the balance of computing load;

[0015] If the partition results are the same, the operator is moved to the corresponding partition area.

[0016] In some embodiments, the method for analyzing the computation time of an operator specifically includes:

[0017] The actual calculation time of the operator is obtained by taking the average of multiple iterations; in the reverse calculation process, if the recalculation technology is used, the forward calculation process of the corresponding operator needs to be re-executed during the error back propagation process.

[0018] In some embodiments, the method for obtaining the transfer cost of an operator specifically includes:

[0019] Calculate the space occupied by the operator based on the data dimension corresponding to the operator;

[0020] Divide the operator data size by the theoretical PCI-E bandwidth to obtain the operator transfer cost.

[0021] In some embodiments, the constraints include: operator computability constraints, operator result video memory storage constraints, operator result internal memory storage constraints, operator swap-in constraints, operator swap-out constraints, computation graph execution constraints, recalculation execution constraints, and video memory capacity constraints; specifically, if there is no feasible solution under the current video memory capacity limit, the video memory capacity upper limit is adjusted and the video memory optimization strategy continues to be generated.

[0022] In some embodiments, the method for calculating the load balancing ratio is:

[0023]

[0024] Among them, T max The maximum time for subgraph calculation, T min Minimum time to compute for a subgraph.

[0025] According to a second aspect, a parallel training apparatus based on adaptive recalculation and load partitioning is provided, comprising:

[0026] The computational graph data structure conversion module is used to convert the dynamic computational graph of the deep learning framework into a static computational graph;

[0027] The computation graph partitioning module is used to partition the computation graph;

[0028] A mixed integer programming model generation module is used to construct a mixed integer programming model according to the operator's calculation time, the operator's transfer cost and the constraint conditions;

[0029] A video memory optimization strategy generation module is used to solve the optimization problem and generate a video memory optimization strategy;

[0030] The load balancing rate calculation module is used to evaluate the execution time of each computing subgraph according to the obtained video memory optimization strategy and calculate the load balancing rate.

[0031] According to a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and wherein the processor implements any method described in the first aspect when executing the computer program.

[0032] According to a fourth aspect, a computer-readable medium having a non-volatile program code executable by a processor is provided, wherein the program code causes the processor to execute any method as described in the first aspect.

[0033] The present invention provides a parallel training method, apparatus, device and computer-readable medium based on adaptive recalculation and load partitioning. The goal of the scheduling strategy is to minimize the calculation or execution time, while ensuring that the video memory consumption of the neural network during the calculation process does not exceed the device video memory limit. This paper converts the above problem into an integer programming problem based on the calculation time of the operator and the transfer cost of the operator, and uses a standard linear programming commercial or open source solver to solve it, generate a video memory optimization strategy, and evaluate the execution time of each calculation subgraph for the generated video memory optimization strategy, and calculate the load balancing rate, and determine whether to continue to divide the calculation subgraph according to the size of the load balancing rate. The method provided by the present invention can find the final operator execution sequence, so that the runtime peak video memory consumption of the calculation graph is within the given limit, and the training time of the neural network model is minimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A flow chart of a parallel training method based on adaptive recalculation and load partitioning provided in an embodiment of this specification is shown;

[0036] Figure 2 A schematic diagram of a process framework of a parallel training method based on adaptive recalculation and load partitioning provided in an embodiment of this specification is shown;

[0037] Figure 3 A schematic diagram of a parallel training device based on adaptive recalculation and load partitioning provided in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0038] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0040] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0041] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more.

[0042] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0043] In order to facilitate understanding of the solution provided by the embodiment of the present application, some terms involved in the solution are first briefly introduced.

[0044] Mixed Integer Programming (MIP): is a mathematical optimization technique that combines the characteristics of linear programming (LP) and integer programming (IP). MIP problems usually involve some decision variables, a linear objective function, and a set of linear constraints, in which some or all variables are restricted to integer values. Mixed integer programming is a powerful optimization tool that is suitable for problems that require both discrete and continuous decisions. By building appropriate models and choosing appropriate solvers and techniques, many complex optimization problems in the real world can be effectively solved.

[0045] PyTorch: is a popular open source machine learning framework that provides a flexible and powerful platform for building and training deep learning models and other types of machine learning models, and supports dynamic construction of computational graphs.

[0046] Computational graph: is a data structure used to represent the computational process. It displays the computational steps and data flow in the algorithm in a graphical way. It can conveniently represent the forward propagation (input data starts from the starting point of the graph, passes through a series of operation nodes, and finally reaches the output node) and backward propagation (starting from the output node, calculating the gradient of each node along the reverse direction of the computational graph) process, which is crucial for training neural networks.

[0047] Figure 1 A schematic diagram of a parallel training method based on adaptive recalculation and load partitioning provided in an embodiment of this specification is shown (for the convenience of understanding this solution, the flowchart diagram can be used as an auxiliary reference). Figure 2 ),like Figure 1 As shown, the method comprises the following steps:

[0048] Step S101, converting the dynamic computation graph in the deep learning framework into a static computation graph data structure.

[0049] Convert the dynamic computational graph in the deep learning framework into a static computational graph data structure. The deep learning framework includes but is not limited to PyTorch, JAX, Chainer, TensorFlow 2.x, and Sonnet. The conversion process can be completed by using the functions in the Just-In-Time Compiler (JIT) module. This step belongs to the general technology in this technical field and will not be repeated here.

[0050] Step S102, topologically sort the operator set in the computation graph according to the dependency relationship to construct a computation sequence, and divide the computation sequence to generate a computation subgraph. The operator is the basic block for constructing the computation graph and is used to perform computation tasks.

[0051] The computational graph contains a series of operation nodes, such as convolution, matrix multiplication, activation function, etc. These nodes are connected by edges (representing tensors or data flows) to form a directed acyclic graph (DAG).

[0052] The process of topologically sorting the operator set in the computation graph according to the dependency relationship to construct the computation sequence is as follows: Based on the computation subgraph generated in the previous step, its nodes represent operators and the edges represent the dependency relationship between operators. Count the in-degree of each node (that is, how many edges point to the node), and the nodes without predecessor nodes (that is, nodes with in-degree 0) are the starting point of sorting. Create a queue or stack to save nodes with in-degree 0. Select a node (node ​​with in-degree 0) from the queue or stack, add it to the output list, and remove it from the graph. Check all successor nodes of the node just removed (that is, the nodes pointed to by the removed node) and reduce their in-degree by 1. If the in-degree of a node becomes 0, add the node to the queue or stack. Repeat the above process until the queue or stack is empty. Finally, check whether the output list contains all the nodes in the graph. If there is a loop, it means that the topological sorting is not completed, and you need to check and repair the loop.

[0053] After constructing the calculation sequence, divide the calculation sequence.

[0054] Exemplarily, first, the weight of the calculation sequence is standardized, and the weight of the cost model calculation operator is calculated using the following formula:

[0055] T fcomp (o k ,v i )=T fe (o k ,v i )—T fs (o k ,v i )

[0056] Among them, T fe (o k ,v i ) is the operator o k Calculate the end time of the operation; T fs (o k ,v1) is the operator o k Calculates the start time of the operation.

[0057] Then the data is standardized using the range standardization method, as shown in the following formula:

[0058]

[0059] Represents the weight value of the operator after standardization, S i Represents the weight of the operator, max(S) and min(S) represent the maximum and minimum weights in the calculation sequence, respectively.

[0060] After normalization, all values ​​of sequence S are scaled to the interval [0,1]. After that, the algorithm will perform an initial partition on the sequence, using a balanced partitioning method. Assuming that the neural network model has a total of n operators, and the neural network model is divided into k parts, the number of operators in each subsequence during the initial partition is [n / k]. After the initial partition, the following steps are completed iteratively: (1) The weight sum of each subsequence is calculated using the following formula.

[0061]

[0062] Among them, T comp (o k , v i ) represents a single operator o k ∈s i On device v i The calculation time on .

[0063] (2) Among all subsequences, select the subsequence with the largest weighted sum. Its weighted sum is denoted as max and its index is denoted as p. (3) Among all subsequences, select the subsequence with the smallest weighted sum. Its weighted sum is denoted as min and its index is denoted as q. (4) Compare the sizes of max and min. If the condition max≤min+1 is satisfied at this time, stop the sequence partitioning algorithm and output the partitioning result. If the condition is not satisfied, adjust the boundary of the smallest partition q at this time, and the direction of the boundary movement is the direction of the largest partition. If the adjusted partition boundary happens to be the boundary of the largest partition p at this time, return to step (1) of the sequence partitioning algorithm and restart a new round of iteration. Otherwise, return to step (2) of the sequence partitioning algorithm and restart a new round of iteration.

[0064] In one embodiment, after the calculation sequence is divided, the calculation sequence is further fine-tuned, and the fine-tuning method specifically includes:

[0065] Step S1021, for each operator in the calculation sequence, check whether the partition results of its successor and predecessor nodes are the same as those of the current node;

[0066] Checking that each operator in the computation graph and its predecessor and successor nodes are located in the same partition result can help verify that the partitioning is reasonable and does not cause unnecessary data transfer.

[0067] Step S1022: If the partition results are different, try to move the operator and the operators before or after the topological sorting to the subsequence where the corresponding partition is located to determine whether the communication volume can be reduced while maintaining the balance of computing load;

[0068] Step S1023: If the division results are the same, move the operator to the corresponding division area.

[0069] The advantage of fine-tuning the calculation sequence is that during parallel training, data transmission between different nodes will bring additional time and bandwidth costs. If operators are assigned to different partitions, the data exchange between them will cause a lot of communication overhead. Therefore, it is very important to ensure that the computing load of each node is distributed as evenly as possible. Unbalanced load will cause some nodes to be overloaded, while other nodes may be idle, thus affecting the overall training efficiency. By reducing unnecessary data transmission and balancing the computing load, the training time of the entire model can be significantly shortened.

[0070] Step S103, constructing a mixed integer programming model according to the calculation time of the operator, the transfer cost of the operator and the constraint conditions. Specifically, the calculation time of the operator in the mixed integer programming model reflects the computing performance of parallel training and video memory optimization, and the transfer cost of the operator reflects the communication performance of the hardware.

[0071] Constructing a mixed integer programming model to optimize the scheduling of operators requires taking the operator's computing time and the operator's transfer cost as decision variables, minimizing the total processing time and transfer cost, and satisfying a series of constraints.

[0072] In one embodiment, the method for analyzing the calculation time of an operator specifically includes: obtaining the actual calculation time of the operator by taking the average of multiple iterations. In the reverse calculation process, if the recalculation technology is used, the forward calculation process of the corresponding operator needs to be re-executed during the error back propagation process.

[0073] Exemplarily, the forward computation time of an operator can be expressed by the following formula:

[0074] T fcomp (o k ,v i )=T fe (o k ,v i )—T fs (o k ,v i )

[0075] Among them, T fe (o k ,v i ) is the operator o k Calculate the end time of the operation; T fs (o k ,v i ) is the operator o k Calculates the start time of the operation.

[0076] Understanding the computation time of each operator helps to develop optimization strategies. In parallel computing, we can decide which operators are suitable for parallelization and how to divide the computing tasks based on the computation time of the operators, thereby maximizing the benefits of parallelization.

[0077] In one embodiment, the method for obtaining the transfer cost of an operator specifically includes: calculating the space occupied by the operator according to the data dimension corresponding to the operator, and dividing the data size of the operator by the theoretical PCI-E bandwidth to obtain the operator transfer cost.

[0078] For example, first, determine the dimension and type of data processed by each operator. For example, the size of a two-dimensional tensor is m×n, and each element is a floating point number (such as float32). Then, calculate the size of the data processed by each operator in bytes. Usually, a float32 occupies 4 bytes and a float64 occupies 8 bytes. The space occupied by the operator can be calculated according to the following formula:

[0079] DATE size =m×n×ELEMENT size

[0080] Among them, DATE size Represents the space occupied by the operator; m and n are the dimensions of the operator tensor; ELEMENT size Represents the size of each element.

[0081] Consult the hardware manual or determine the PCI-E bandwidth through experiments. PCI-E bandwidth is usually measured in GB / s (gigabytes per second). Finally, the data size of the operator is DATE size Divide by the PCI-E bandwidth to get the time required for operator transfer.

[0082] During parallel training, GPUs are good at processing large amounts of data in parallel, while CPUs are more suitable for processing control logic and computing tasks with small amounts of data. Computing tasks suitable for CPU processing can be transferred from GPUs to CPUs. Furthermore, some computationally intensive tasks may be more suitable for execution on GPUs, in which case operators need to be transferred from CPUs to GPUs. In addition, when the GPU's memory usage exceeds the limit, some computing tasks can be transferred from GPUs to CPUs. Therefore, the main purpose of operator transfer is to optimize the utilization of computing resources, improve computing efficiency, and reduce communication overhead.

[0083] In one embodiment, the constraints of the mixed integer programming model include: operator computability constraints, operator result memory storage constraints, operator result internal memory storage constraints, operator swap-in constraints, operator swap-out constraints, computation graph execution constraints, recalculation execution constraints, and memory capacity constraints. If there is no feasible solution under the current memory capacity limit, the upper limit of the memory capacity is adjusted, and the memory optimization strategy is continued to be generated.

[0084] The setting of constraints for the mixed integer programming model can ensure that the model's solution meets actual requirements and business rules.

[0085] Step S104: using a solver to solve the scheduling optimization problem of operators in the mixed integer programming model to generate a video memory optimization strategy.

[0086] Use standard commercial or open source linear programming solvers, including but not limited to CPLEX, Gurobi, SCIP or Xpress, to solve the scheduling optimization problem of operators in the mixed integer programming model. Before solving, you can set some solution parameters, such as solution time limit, optimization goal (such as minimization or maximization), and heuristic algorithm parameters. The solver will finally provide one or more solutions, including the optimal solution and possible suboptimal solutions.

[0087] The weight data and gradient data of the neural network model are always stored in the video memory, and the object of transfer is the output activation value of the operator. The goal of its video memory optimization strategy is to design the optimal operation execution order without exceeding the video memory capacity limit, thereby reducing the peak video memory usage during the calculation process and shortening the training time of the entire neural network model as much as possible.

[0088] Exemplarily, Table 1 is a calculation result of an operator scheduling optimization problem in a mixed integer programming model, as shown in the following table:

[0089]

[0090] Table 1

[0091] Step S105 , evaluating the execution time of each computing subgraph according to the generated video memory optimization strategy, and calculating the load balancing rate.

[0092] The evaluation result of the execution time of each computation subgraph can be obtained according to the calculation method of the operator's computation time and the operator's transfer cost in step S103, and this step will not be repeated. Finally, the load balancing rate of the computation subgraph can be calculated according to the evaluation result.

[0093] In one embodiment, the calculation formula for calculating the load balancing rate of the subgraph is:

[0094]

[0095] Among them, T max The maximum time to calculate the subgraph; T min The shortest time for subgraph calculation; the load balancing rate is between 0 and 1, and the larger the value, the better the load balancing degree.

[0096] Step S106, determining the final divided computing subgraph according to the calculation result of the load balancing rate, and using the final divided computing subgraph to perform parallel training.

[0097] In one embodiment, the final divided computation subgraph can be determined by judging whether the load balancing rate meets the set threshold. If the set threshold is met, the computation subgraph divided this time is determined to be the final divided computation subgraph. If not, the computation graph is continued to be divided until the load balancing rate meets the set threshold. After the threshold is met, the computation subgraph divided this time is determined to be the final divided computation subgraph. Finally, the final divided computation subgraph is used for parallel training.

[0098] Exemplarily, the threshold of the load balancing rate is set to 0.7. If the load balancing rate L is greater than 0.7, the current divided computational subgraph is determined to be the final divided computational subgraph. If the load balancing rate L is less than or equal to 0.7, the computational graph is continued to be divided, and this cycle is repeated until the load balancing rate is greater than 0.7. After the load balancing rate L is greater than 0.7, the computational subgraph of this division is determined to be the final divided computational subgraph. Finally, the final divided computational subgraph is used for parallel training.

[0099] Corresponding to the above method provided by the present invention, the present invention also provides a device. Figure 3 FIG. 2 shows a schematic diagram of a parallel training device based on adaptive recalculation and load division provided in an embodiment of the present specification. Figure 3 As shown, the device comprises:

[0100] The computation graph data structure conversion module S301 is used to convert the dynamic computation graph of the deep learning framework into a static computation graph.

[0101] The computation graph partitioning module S302 is used to partition the computation graph.

[0102] The mixed integer programming model generation module S303 constructs a mixed integer programming model according to the calculation time of the operator, the transfer cost of the operator and the constraint conditions.

[0103] The video memory optimization strategy generation module S304 is used to solve the optimization problem and generate a video memory optimization strategy.

[0104] The load balancing rate calculation module S305 is used to evaluate the execution time of each computing subgraph according to the obtained video memory optimization strategy, and calculate the load balancing rate.

[0105] It is worth mentioning that Figure 3 The device shown is Figure 1 The method embodiments shown correspond to and can be applied to Figure 1 The calculation graph data structure conversion, calculation graph partitioning, mixed integer programming model construction, video memory measurement generation, and load balancing rate calculation in the illustrated method embodiment are not described in detail here. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0106] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 1 The method described.

[0107] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 1 The method described.

[0108] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0109] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A parallel training method based on adaptive recomputation and load partitioning, characterized in that: The method comprises: Convert the dynamic computation graph in the deep learning framework into a static computation graph data structure; specifically, the static computation graph has a fixed data structure; The operator set in the computation graph is topologically sorted according to the dependency relationship to construct a computation sequence, and the computation sequence is divided to generate a computation subgraph; the operator is the basic block for constructing the computation graph and is used to perform computation tasks; A mixed integer programming model is constructed based on the operator's computation time, the operator's transfer cost, and the constraints. Specifically, the operator's computation time in the mixed integer programming model reflects the computational performance of parallel training, and the operator's transfer cost reflects the communication performance of the hardware. Use the solver to solve the scheduling optimization problem of operators in the mixed integer programming model to generate a video memory optimization strategy; Based on the generated video memory optimization strategy, the execution time of each computational subgraph is evaluated and the load balancing rate is calculated. The final divided computational subgraph is determined according to the calculation result of the load balancing rate, and the final divided computational subgraph is used for parallel training.

2. The method according to claim 1, characterized in that: After the calculation sequence is divided, the calculation sequence is fine-tuned. The fine-tuning method specifically includes: For each operator in the calculation sequence, check whether the partition results of its successor and predecessor nodes are the same as those of the current node; If the partition results are different, try to move the operator and the operators before or after the topological sorting to the subsequence where the corresponding partition is located to determine whether the communication volume can be reduced while maintaining the balance of computing load; If the partition results are the same, the operator is moved to the corresponding partition area.

3. The method according to claim 1, characterized in that The method of analyzing the calculation time of an operator specifically includes: The actual calculation time of the operator is obtained by taking the average of multiple iterations; in the reverse calculation process, if the recalculation technology is used, the forward calculation process of the corresponding operator needs to be re-executed during the error back propagation process.

4. The method according to claim 1, characterized in that: The method for obtaining the transfer cost of an operator includes: Calculate the space occupied by the operator based on the data dimension corresponding to the operator; Divide the operator data size by the theoretical PCI-E bandwidth to obtain the operator transfer cost.

5. The method according to claim 1, characterized in that The constraints include: operator computability constraints, operator result memory storage constraints, operator result internal memory storage constraints, operator swap-in constraints, operator swap-out constraints, computation graph execution constraints, recalculation execution constraints, and memory capacity constraints. Specifically, if there is no feasible solution under the current memory capacity limit, the upper limit of the memory capacity is adjusted and the memory optimization strategy continues to be generated.

6. The method according to claim 1, characterized in that The calculation method for calculating the load balancing rate is: Among them, T max The maximum time for subgraph calculation, T min Minimum time to compute for a subgraph.

7. A parallel training device based on adaptive recalculation and load partitioning, characterized in that: The method according to claim 1 is used to perform parallel training of adaptive recomputation and load partitioning, wherein the apparatus comprises: The computational graph data structure conversion module is used to convert the dynamic computational graph of the deep learning framework into a static computational graph; The computation graph partitioning module is used to partition the computation graph; A mixed integer programming model generation module is used to construct a mixed integer programming model according to the operator's calculation time, the operator's transfer cost and the constraint conditions; A video memory optimization strategy generation module is used to solve the optimization problem and generate a video memory optimization strategy; The load balancing rate calculation module is used to evaluate the execution time of each computing subgraph according to the obtained video memory optimization strategy and calculate the load balancing rate.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 6.

Images