Model training method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention discloses a model training method and device, electronic equipment and a readable storage medium. The method comprises the steps of generating an initial calculation graph according to a training script of model training; when the calculation graph is issued, performing memory static analysis on the initial calculation graph to obtain a calculation graph memory analysis result; segmenting the initial calculation graph according to the calculation graph memory analysis result and the residual running memory of a model training actuator to obtain a plurality of candidate calculation graphs; and compiling each candidate calculation graph to generate an executable file for model training. According to the method, compiling optimization is carried out based on a computational graph mode, the executable file is generated in advance, the model training efficiency can be improved, the computational graph is adaptively segmented according to the remaining running memory of the actuator, the running problem of the computational graph can be effectively solved, and the stability of model training is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and readable storage medium for model training. Background Art

[0002] With the rise of artificial intelligence generated content (AIGC) technology, large language models have developed rapidly. The model size of large language models has rapidly increased from hundreds of millions of parameters to hundreds of billions of parameters. While improving the model's prediction accuracy, it has also brought great challenges to the model training executor. For example, more parameters mean more computing resources and storage space.

[0003] In traditional model training, improving the training efficiency of large language models usually requires higher computing and storage resources, and often cannot take into account both the improvement of computing efficiency and the efficient use of memory. Especially in the training of large language models, if the current storage status cannot run the training program, the program will be directly exited, resulting in the inability of model training to run stably. Summary of the invention

[0004] The present invention provides a method, device, electronic device and readable storage medium for model training to improve the efficiency of model training and ensure the stability of model training.

[0005] According to one aspect of the present invention, a model training method is provided, the method comprising:

[0006] Generate an initial computational graph based on the training script for model training;

[0007] When the computation graph is issued, a memory static analysis is performed on the initial computation graph to obtain a computation graph memory analysis result;

[0008] According to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is segmented to obtain multiple candidate calculation graphs;

[0009] Compile each candidate computational graph to generate an executable file for model training.

[0010] According to another aspect of the present invention, a model training device is provided, the device comprising:

[0011] The initial calculation graph generation module is used to generate the initial calculation graph according to the training script of the model training;

[0012] A memory static analysis module is used to perform memory static analysis on the initial calculation graph when the calculation graph is issued, and obtain the calculation graph memory analysis result;

[0013] A computation graph segmentation module, used to segment the initial computation graph according to the computation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate computation graphs;

[0014] The computational graph compilation module is used to compile each candidate computational graph and generate an executable file for model training.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model training method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model training method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, which implements the model training method described in any embodiment of the present invention when executed by a processor.

[0021] The technical solution of the embodiment of the present invention generates an initial calculation graph according to a training script for model training; when the calculation graph is issued, a memory static analysis is performed on the initial calculation graph to obtain a calculation graph memory analysis result; according to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is segmented to obtain multiple candidate calculation graphs; each candidate calculation graph is compiled to generate an executable file for model training, which solves the problem of low efficiency and poor stability of model training, can ensure that the memory of the executor is fully utilized, and ensures the stability of model training while improving the efficiency of model training.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a model training method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 This is a schematic diagram of generating a computational graph provided according to Embodiment 1 of the present invention;

[0026] Figure 3 It is a schematic diagram of a process of generating an executable file according to a calculation graph provided in accordance with Embodiment 1 of the present invention;

[0027] Figure 4 is a flowchart of a model training method provided according to Embodiment 2 of the present invention;

[0028] Figure 5 This is a schematic diagram of memory static analysis of a computation graph provided according to Embodiment 2 of the present invention;

[0029] Figure 6 This is an application example diagram of a model training method provided in Embodiment 2 of the present invention;

[0030] Figure 7 is a flowchart of a model training method in a warm-up phase provided according to Embodiment 2 of the present invention;

[0031] Figure 8 is a flow chart of a model training method in a stable phase provided according to Embodiment 2 of the present invention;

[0032] Fig. 9 is a structural schematic diagram of a model training device provided according to Embodiment 3 of the present invention;

[0033] Fig.10 It is a schematic diagram of the structure of an electronic device for implementing the model training method of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment 1

[0037] Figure 1 This is a flow chart of a model training method provided according to the first embodiment of the present invention. This embodiment is applicable to the case where the training of a neural network model is accelerated and the actuator memory is fully utilized. The method can be executed by a model training device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device such as a computer. Figure 1 As shown, the method includes:

[0038] Step 110: Generate an initial calculation graph according to the training script of the model training.

[0039] The calculation graph is a graphical structure used to represent the calculation steps of the model algorithm. The initial calculation graph can be obtained by collecting node information in the model algorithm, converting the user-level calculation into an intermediate-level calculation expression, and then generating a connection graph between algorithm nodes based on the intermediate calculation expression when the calculation expression needs to be executed.

[0040] For example, Figure 2 It is a schematic diagram of generating a computational graph according to the first embodiment of the present invention. For example, the model script in model training generally includes: defining a model file, defining an optimizer, and starting a training loop iteration. The iterative training includes: reading data, forward calculation, calculating a loss function (loss), reverse calculation, and the optimizer updating weights to complete a batch of data training. The computational graph can be generated based on the forward calculation and reverse calculation in the model script. Figure 2As shown in the figure, the computational graph may include computational nodes and edges between computational nodes. The computational node information includes but is not limited to the properties of tensors in the model algorithm, such as shape, layout, data type, storage in high bandwidth memory (HBM), and intermediate representation definition operations (IR build op).

[0041] In the embodiment of the present invention, the computation graph may also be optimized, which may include but is not limited to optimization measures such as constant folding and dead code elimination.

[0042] In order to make users unaware when generating calculation graphs, calculation graphs can be generated at specific times to avoid additional time consumption for model training.

[0043] In an optional implementation of an embodiment of the present invention, an initial calculation graph is generated according to a training script of model training, including: determining a time to generate the calculation graph according to script execution operations and / or user instructions in model training; at the time of calculation graph generation, generating an initial calculation graph according to the training script of model training.

[0044] The timing of calculating graph generation determined by the script execution operation may be when the executor occupancy is low during model training, and the calculation graph generation can be hidden. For example, the timing of calculating graph generation determined by the script execution operation may be when the model training hardware does not support the need to send the processor to execute (the user calls the to cpu operation), or when the optimizer is updated (optimizer).

[0045] The timing of calculating graph generation determined according to user instructions may be a timing when calculating graph generation can be hidden in user monitoring operations. For example, the timing of calculating graph generation determined according to user instructions may be when the user calls debug printing (print), or when the user joins the graph distribution operation.

[0046] By generating a computational graph when a user calls the to cpu operation, when the optimizer is updated, when a user calls debug printing, or when a user joins the graph distribution operation, the user can be unaware of the computational graph generation, thus avoiding additional consumption for model training.

[0047] Step 120: When the computation graph is sent down, a static memory analysis is performed on the initial computation graph to obtain a computation graph memory analysis result.

[0048] The received computation graph can be subjected to a memory static analysis. The memory static analysis can estimate the memory size required for each computation node based on the input and output information of each computation node in the computation graph. In the memory static analysis, it can also be considered whether the memory occupied by each computation node can be recycled and reused, so as to accurately estimate the memory required for the computation graph to be executed when the computation graph is not executed (i.e., static analysis).

[0049] Among them, when considering whether the memory can be recovered and reused, it can be determined whether the current computing node is still used by subsequent computing nodes based on the connection relationship of the computing nodes in the computing graph; if it is still used, the memory of the current computing node cannot be recovered and reused; if it is not used, the memory of the current computing node can be recovered and reused.

[0050] By performing static memory analysis on the computation graph, the memory required for the computation graph can be estimated before the computation graph is executed. Based on the results of the computation graph memory analysis, it can be determined whether the executor can meet the computation requirements of the computation graph.

[0051] Step 130: According to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is split to obtain multiple candidate calculation graphs.

[0052] When the remaining running memory of the executor can meet the results of the calculation graph memory analysis, the calculation graph can be directly compiled to generate an executable file for model training to execute the calculation graph and perform model training. However, when the remaining running memory of the executor cannot meet the results of the calculation graph memory analysis, in order to ensure the full utilization of the executor and avoid reporting an error and exiting the program due to insufficient memory, the calculation graph can be split so that it can be executed on the executor.

[0053] There are many ways to split the computation graph. For example, you can use the binary method to split the computation graph. Figure 1 Divide into two parts, and then perform memory static analysis on the calculation graph obtained after segmentation to determine whether it meets the remaining running memory of the executor; if not, continue to segment. If it meets the requirements, the calculation graph after segmentation can be compiled.

[0054] For another example, during the memory static analysis process, the analysis can be performed while the graph is being segmented. Exemplarily, during the memory static analysis, when the memory analysis result of the current computation graph is about to reach the remaining running memory of the executor, the computation graph is segmented. After segmentation, a portion of the computation graph can be executed on the executor; the memory static analysis is continued for the remaining computation graph, and when the remaining running memory of the executor is about to be reached, the computation graph is segmented. Repeat this process until the entire computation graph is analyzed and all the segmented parts can be executed on the executor.

[0055] In an optional implementation of an embodiment of the present invention, the initial computation graph is segmented according to the computation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate computation graphs, including: using a binary search method to segment the initial computation graph to obtain a current candidate computation graph; and performing a memory static analysis on the current candidate computation graph; when the memory analysis result of the current candidate computation graph is greater than the remaining running memory of the model training executor, using a binary search method to segment the current candidate computation graph to obtain a next candidate computation graph; updating the next candidate computation graph to the current candidate computation graph, and returning to the memory static analysis step of the current candidate computation graph until the memory analysis result of the current candidate computation graph is less than or equal to the remaining running memory of the model training executor.

[0056] Exemplarily, the pseudo code for splitting the computation graph may be:

[0057] left=0,right=nodes_num-1 / / left is the first calculation node in the current calculation graph, right is the last calculation node

[0058] while(left<=right){

[0059] mid=(left+right) / 2 / / Calculate the middle node

[0060] mem[mid]=memanalysis(mid) / / Perform static memory analysis on the intermediate node

[0061] if(mem[mid]>=target) / / Memory static analysis result is greater than or equal to the remaining running memory

[0062] right=mid-1 / / Split the current calculation graph to get the calculation graph from left to right

[0063] else

[0064] left=mid+1 / / Analyze the next calculation graph

[0065] }

[0066] return right

[0067] The computational graph is split by using the binary method, which is simple and does not bring additional time overhead. It ensures that the executor can meet the memory required for the computational graph, thereby avoiding the exit of model training due to insufficient memory, improving the reliability of model training, and making full use of the executor memory.

[0068] Step 140: compile each candidate computation graph to generate an executable file for model training.

[0069] Figure 3 FIG. 1 is a flow chart of generating an executable file based on a calculation graph according to Embodiment 1 of the present invention. Figure 3 As shown in the figure, computing nodes are defined in the model algorithm. The computing nodes are used as the conversion entry to convert the model algorithm into an intermediate expression. The computing graph can be obtained based on the intermediate expression result. The computing graph can be optimized to generate an executable file. The algorithm can be executed to implement model training by executing the executable file.

[0070] In order to make the user unaware of the generation of the executable file and avoid extra time consumption for model training, in an optional implementation of the embodiment of the present invention, each candidate calculation graph is compiled to generate an executable file for model training, including: determining the timing of calculation graph compilation according to the script execution operation and / or user instructions in the model training; at the calculation graph compilation timing, compiling each candidate calculation graph to generate an executable file for model training.

[0071] Specifically, determining the timing of graph compilation based on the script execution operation in model training can be determining the timing of graph compilation when the model training is not supported by the hardware and needs to be sent to the processor for execution, or when the optimizer is updated. Determining the timing of graph compilation based on user instructions in model training can be determining the timing of graph compilation when the model training is called by the user for debugging printing, or when the user joins the graph distribution operation.

[0072] The technical solution of this embodiment generates an initial calculation graph according to the training script of model training; when the calculation graph is issued, the initial calculation graph is subjected to static memory analysis to obtain the calculation graph memory analysis result; according to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is segmented to obtain multiple candidate calculation graphs; each candidate calculation graph is compiled to generate an executable file for model training, which solves the problems of low efficiency and poor stability of model training. By segmenting the calculation graph, it can be ensured that the memory of the executor is fully utilized, and the stability of model training is guaranteed while improving the efficiency of model training; by pre-generating the calculation graph and the corresponding executable file when the calculation graph is issued, the efficiency of model training can be improved, and the use of the model graph can be imperceptible to users.

[0073] Embodiment 2

[0074] Figure 4 It is a flowchart of a model training method provided according to the second embodiment of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments.

[0075] Optionally, when the computation graph is issued, a static memory analysis is performed on the initial computation graph to obtain a computation graph memory analysis result, including: topologically sorting the initial computation graph to obtain an operator execution sequence; determining the memory size occupied by each computing node based on the operator input information and operator output information in the operator execution sequence; determining whether the memory statically allocated to the target computing node can be reclaimed based on the reference relationship between the computing nodes in the operator execution sequence; when performing a static memory analysis on subsequent computing nodes of the target computing node, allocating memory to subsequent computing nodes based on the reclaimed memory; and determining the computation graph memory analysis result based on the static memory allocation result of each computing node and the corresponding memory size.

[0076] like Figure 4 As shown, the method includes:

[0077] Step 410: Generate an initial computation graph according to the training script of the model training.

[0078] Optionally, an initial calculation graph is generated according to a training script of model training, including: determining a time to generate the calculation graph according to script execution operations and / or user instructions in model training; and generating an initial calculation graph according to the training script of model training at the time of calculation graph generation.

[0079] The timing of generating the computational graph determined by the script execution operation can be when the model training needs to be sent to the processor for execution because the hardware does not support it, or when the optimizer is updated. The timing of generating the computational graph determined by the user instruction can be when the user calls the debug print, or when the user joins the graph distribution operation.

[0080] Step 420: topologically sort the initial computation graph to obtain an operator execution sequence.

[0081] For example, Figure 5 Schematic diagram of memory static analysis of a computation graph provided according to Embodiment 2 of the present invention. Figure 5 As shown, the operator execution sequence can be: computing node A->computing node B->computing node C->computing node D->computing node G, and computing node A->computing node E->computing node F->computing node G.

[0082] Step 430: Determine the memory size occupied by each computing node according to the operator input information and operator output information in the operator execution sequence.

[0083] Step 440: Determine whether the memory statically allocated to the target computing node can be reclaimed based on the reference relationship between computing nodes in the operator execution sequence.

[0084] The reference relationship may be the reference count of a computing node by subsequent computing nodes. Figure 5As shown, initially, computing node A is referenced by computing node B and computing node E, and the reference count of computing node A is 2. When analyzing different computing nodes, the reference count changes dynamically. For example, when analyzing computing node C, the reference count of computing node A is 0. In an embodiment of the present invention, when a computing node is not referenced by a subsequent computing node, such as when the reference count is 0, the memory of the computing node can be reclaimed.

[0085] In an optional implementation of an embodiment of the present invention, it is determined whether the memory statically allocated to the target computing node can be reclaimed based on the reference relationship between computing nodes in the operator execution sequence, including: marking the reference count of each computing node referenced by other computing nodes according to the operator execution sequence, and adjusting the reference count based on the static analysis of the memory of each computing node; when the reference count of the target computing node is zero, the memory occupied by the target computing node is reclaimed.

[0086] Step 450: When performing memory static analysis on subsequent computing nodes of the target computing node, allocate memory to the subsequent computing nodes according to the recovered memory.

[0087] The reclaimed memory can be put into the released memory pool. When allocating memory to a computing node, you can choose between the released memory pool and the newly allocated memory to allocate memory for subsequent computing nodes. For example, Figure 5 The computing node C in can reuse the memory M_A released by the computing node A.

[0088] Step 460: Determine the computation graph memory analysis result based on the static memory allocation result of each computing node and the corresponding memory size.

[0089] When performing static memory allocation, the reusability of memory between computing nodes is considered. When reusing memory, the reusing memory size can be set to the maximum value of the memory sizes occupied by multiple reusing computing nodes.

[0090] When performing static memory analysis, you can also consider the parallelism between computing nodes to avoid dependencies that affect efficiency. For example, when considering parallelism, Figure 5 The computing node F cannot reuse the memory M_C of the computing node C to avoid the computing node F having to wait for the computing node C to complete, which affects the efficiency.

[0091] In static memory analysis, if there are multiple compute nodes whose memory can be reused for the current compute node, the compute node with the closest memory can be selected for reuse based on the memory size occupied by each compute node. For example, Figure 5Compute node G can reuse the memory of compute node C and compute node E. If the memory occupied by compute node C is closest to the memory occupied by compute node G, compute node G reuses the memory M_C of compute node C.

[0092] In order to explain in detail the memory static analysis method provided by the embodiment of the present invention, Figure 5 As shown, an example of memory static analysis application is: for operator execution sequence: computing node A->computing node B->computing node C->computing node D->computing node G, and computing node A->computing node E->computing node F->computing node G. In step 1, the reference count of each computing node is initialized to the reference count of computing node A is 2, the reference count of computing node B is 1, the reference count of computing node C is 1, the reference count of computing node D is 1, the reference count of computing node E is 1, the reference count of computing node F is 1, and the reference count of computing node G is 1.

[0093] In step 2, memory M_A is allocated to computing node A. In step 3, memory M_B is allocated to computing node B. At this time, the reference count of computing node A is not 0, so computing node B cannot reuse the memory of computing node A. In step 4, considering parallelism, memory M_E is allocated to computing node E. At this time, the reference count of computing node A is 0, and the memory M_A of computing node A can be recycled, such as putting M_A into the released memory pool.

[0094] In step 5, memory M_C is allocated to computing node C. At this time, memory M_A exists in the released memory pool and can be reused. Therefore, memory M_C can reuse M_A. When memory M_C is allocated to computing node C, the reference count of computing node B is 0, and memory M_B can be recycled, such as putting M_B into the released memory pool.

[0095] In step 6, memory M_D is allocated to computing node D. At this time, memory M_B exists in the released memory pool and can be reused. Therefore, memory M_D can reuse M_B. When memory M_D is allocated to computing node D, the reference count of computing node C is 0, and memory M_C can be recycled, such as putting M_C into the released memory pool.

[0096] In step 7, memory M_F is allocated to computing node F. Although memory M_C exists in the memory pool released in step 7, M_F does not reuse M_C considering the parallelism of computation graph execution. Specifically, in parallel execution, it is impossible to know Figure 5The execution rates of the left and right sides are fast or slow. When M_F reuses M_C, there is a dependency between the two. The computing node F needs to wait for the computing node C to complete the calculation before the memory can be reused, which affects the efficiency. Therefore, in the embodiment of the present invention, considering the parallelism, M_F does not reuse M_C. When the memory M_F is allocated to the computing node F, the reference count of the computing node E is 0, and the memory M_E can be recycled, such as putting M_E into the released memory pool.

[0097] In step 8, memory M_G is allocated to computing node G. At this time, memory M_C and M_E are available for reuse in the released memory pool. Considering the proximity of memory occupancy, M_C is closest to M_G, and M_G can reuse memory M_C.

[0098] In step nine, a static memory analysis is performed, and the final result of the memory analysis of the computational graph is: M_A is allocated to computing node A, M_B is allocated to computing node B, computing node C reuses M_A, computing node D reuses M_B, computing node E is allocated M_E, computing node F is allocated M_F, and computing node G reuses M_C, that is, M_A is reused. Considering that the reused memory size is set to the maximum value of the memory sizes occupied by multiple reused computing nodes, the memory size of the reused memory M_A is set to the maximum value of the memory sizes occupied by computing nodes A, C, and G.

[0099] By performing static memory analysis on the computation graph and considering the reference count of the computation node by the subsequent computation nodes, the parallelism of the computation graph execution, the proximity between the reused memories, and the maximum memory size occupied by multiple reused computation nodes when selecting the reused memory, the rationality and accuracy of the static memory analysis of the computation graph can be improved, so that the static analysis results are consistent with the actual memory usage and the executableness of the computation graph on the executor is improved.

[0100] Step 470: According to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is segmented to obtain multiple candidate calculation graphs.

[0101] Optionally, the initial computation graph is segmented according to the computation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate computation graphs, including: segmenting the initial computation graph using a binary search method to obtain a current candidate computation graph; and performing a memory static analysis on the current candidate computation graph; when the memory analysis result of the current candidate computation graph is greater than the remaining running memory of the model training executor, segmenting the current candidate computation graph using a binary search method to obtain a next candidate computation graph; updating the next candidate computation graph to the current candidate computation graph, and returning to the memory static analysis step for the current candidate computation graph until the memory analysis result of the current candidate computation graph is less than or equal to the remaining running memory of the model training executor.

[0102] Alternatively, the initial computation graph is segmented according to the computation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate computation graphs, including: when analyzing the current computation node in the computation graph, if it is determined that the computation graph memory analysis result from the starting computation node to the current computation node satisfies the requirement of being greater than the remaining running memory of the model training executor, then segmenting is performed at the previous computation node of the current computation node to obtain a candidate computation graph consisting of the starting computation node to the previous computation node; and the current computation node is updated to the starting computation node, memory static analysis is performed, and the judgment process of the computation graph memory analysis result and the remaining running memory is repeated until the entire computation graph is analyzed to obtain multiple candidate computation graphs.

[0103] Step 480: compile each candidate computation graph to generate an executable file for model training.

[0104] Optionally, an initial calculation graph is generated according to the training script of the model training, including: determining the timing of compiling the calculation graph according to the script execution operations and / or user instructions in the model training; at the timing of compiling the calculation graph, compiling each candidate calculation graph to generate an executable file for the model training.

[0105] The timing of graph compilation determined by script execution can be when model training needs to be sent to the processor for execution when the hardware does not support it, or when the optimizer is updated. The timing of graph compilation determined by user instructions can be when the user calls debug printing, or when the user joins the graph distribution operation.

[0106] Based on the above implementation, optionally, after compiling each candidate calculation graph to generate an executable file for model training, it also includes: determining a hash value of each candidate calculation graph based on node information of each calculation node in each candidate calculation graph; and caching the executable file of the corresponding candidate calculation graph into a cache for model training based on each hash value.

[0107] Among them, the node information includes but is not limited to: input, output, shape, data type, and layout. According to the node information of the computing node, a corresponding node hash value can be generated for each computing node. According to the hash value of each node, a unique hash value can be generated for the computing graph. When caching the candidate computing graph, it can be cached according to the hash value of the computing graph. When searching for the computing graph in the cache, the search method is simple and there is no need to compare the computing nodes one by one, thereby improving the execution efficiency.

[0108] Optionally, the method further includes: querying a cache for model training according to a target hash value of a currently issued computation graph, obtaining an executable file of a target candidate computation graph corresponding to the target hash value, and performing model training.

[0109] By caching the computation graph and the corresponding executable file in the cache, in the subsequent computation graph execution, the hash value comparison is directly performed in the cache to determine the executable file of the target candidate computation graph corresponding to the currently issued computation graph, and the model training is performed. There is no need to repeat the computation graph memory static analysis, computation graph segmentation and computation graph compilation, which can improve the execution efficiency.

[0110] Figure 6 FIG. 1 is an application example diagram of a model training method provided according to Embodiment 2 of the present invention. Figure 6 As shown, when executing the model code during model training, a calculation graph can be generated according to the model algorithm; when the graph is sent down for execution, first check whether there is a cached calculation graph in the cache according to the hash value; if it exists, obtain the executable file of the corresponding calculation graph in the cache for sending down for execution. If it does not exist, perform memory static analysis and calculation graph segmentation on the calculation graph, and then compile the calculation graph according to the segmentation result to obtain an executable file for sending down for execution. For the compiled executable file, the hash value of the calculation graph can be calculated, and the executable file corresponding to the calculation graph can be cached in the cache according to the hash value.

[0111] In practical applications, model training is a process of continuous iterative calculations. The model training phase can be divided into a warm-up phase (warm-up iteration) and a stable training phase. In the warm-up phase, calculation graph generation, calculation graph segmentation, calculation graph compilation, and executable file caching can be completed. In the stable training phase, calculations can be sent down based on the cached results of the warm-up phase, without the need for repeated compilation, and model training can be performed efficiently.

[0112] Figure 7 : is a flow chart of a model training method in the warm-up phase provided according to the second embodiment of the present invention. Figure 7 As shown in the figure, when executing the model code in the model training in the warm-up phase, a calculation graph can be generated according to the model algorithm; when the graph is sent down for execution, the calculation graph is statically analyzed in memory and segmented, and then the calculation graph is compiled according to the segmentation results to obtain an executable file, which is then sent down for execution. For the compiled executable file, the hash value of the calculation graph can be calculated, and the executable file corresponding to the calculation graph can be cached in the cache according to the hash value.

[0113] Figure 8 FIG. 1 is a flow chart of a model training method in a stable phase according to Embodiment 2 of the present invention. Figure 8As shown in the figure, when executing the model code in the model training in the stable stage, when sending the graph for execution, the cache is checked according to the hash value to see if there is a cached calculation graph in the cache; if so, the executable file of the corresponding calculation graph is obtained in the cache for sending and execution. If not, the calculation graph is compiled to obtain an executable file for sending and execution.

[0114] The technical solution of the embodiment of the present invention generates an initial calculation graph according to a training script for model training; topologically sorts the initial calculation graph to obtain an operator execution sequence; determines the memory size occupied by each computing node according to the operator input information and the operator output information in the operator execution sequence; determines whether the memory statically allocated to the target computing node can be recycled according to the reference relationship between the computing nodes in the operator execution sequence; when performing memory static analysis on the subsequent computing nodes of the target computing node, allocates memory to the subsequent computing nodes according to the recycled memory; determines the calculation graph memory analysis result according to the static memory allocation result of each computing node and the corresponding memory size; divides the initial calculation graph according to the calculation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate calculation graphs; compiles each candidate calculation graph to generate an executable file for model training, which solves the problem of low efficiency and poor stability of model training, can make efficient use of memory through calculation graph memory static analysis and calculation graph segmentation, and can make users unaware by generating and compiling the calculation graph at an appropriate time, thereby improving computing efficiency, and can reduce repeated compilation of the calculation graph through an efficient caching mechanism to improve model training efficiency.

[0115] Embodiment 3

[0116] Fig. 9 Schematic diagram of a model training device according to Embodiment 3 of the present invention. Fig. 9 As shown, the device includes: an initial calculation graph generation module 910, a memory static analysis module 920, a calculation graph segmentation module 930 and a calculation graph compilation module 940. Among them:

[0117] An initial calculation graph generation module 910 is used to generate an initial calculation graph according to a training script for model training;

[0118] The memory static analysis module 920 is used to perform memory static analysis on the initial calculation graph when the calculation graph is issued, and obtain the calculation graph memory analysis result;

[0119] A computation graph segmentation module 930 is used to segment the initial computation graph according to the computation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate computation graphs;

[0120] The computation graph compilation module 940 is used to compile each candidate computation graph to generate an executable file for model training.

[0121] Optionally, the initial computation graph generation module 910 includes:

[0122] A calculation graph generation timing determination unit, used to determine the calculation graph generation timing according to the script execution operation and / or user instructions in the model training;

[0123] The initial calculation graph determination unit is used to generate an initial calculation graph according to a training script of model training when the calculation graph is generated.

[0124] Optionally, the memory static analysis module 920 includes:

[0125] An operator execution sequence determination unit is used to topologically sort the initial computation graph to obtain an operator execution sequence;

[0126] A node occupied memory size determination unit, used to determine the memory size occupied by each computing node according to operator input information and operator output information in an operator execution sequence;

[0127] A node recycling judgment unit is used to determine whether the memory statically allocated to the target computing node can be recycled according to the reference relationship between computing nodes in the operator execution sequence;

[0128] A memory allocation unit, used to allocate memory to subsequent computing nodes according to the recovered memory when performing memory static analysis on subsequent computing nodes of the target computing node;

[0129] The memory analysis result determination unit is used to determine the memory analysis result of the computation graph according to the static memory allocation result of each computing node and the corresponding memory size.

[0130] Optionally, the node recycling judgment unit is specifically used to:

[0131] According to the operator execution sequence, mark the reference count of each computing node referenced by other computing nodes, and adjust the reference count based on the static memory analysis of each computing node;

[0132] When the reference count of the target computing node is zero, the memory occupied by the target computing node is reclaimed.

[0133] Optionally, the computation graph segmentation module 930 includes:

[0134] The current candidate computation graph determination unit is used to segment the initial computation graph using a binary method to obtain the current candidate computation graph; and perform memory static analysis on the current candidate computation graph;

[0135] A next candidate computation graph determination unit is used to split the current candidate computation graph by a binary method to obtain a next candidate computation graph when the memory analysis result of the current candidate computation graph is greater than the remaining running memory of the model training executor;

[0136] The current candidate computation graph update unit is used to update the next candidate computation graph to the current candidate computation graph, and return to the memory static analysis step for the current candidate computation graph until the memory analysis result of the current candidate computation graph is less than or equal to the remaining running memory of the model training executor.

[0137] Optionally, the device further includes:

[0138] A hash value determination module is used to determine the hash value of each candidate calculation graph according to the node information of each calculation node in each candidate calculation graph after compiling each candidate calculation graph to generate an executable file for model training;

[0139] The executable file cache module is used to cache the executable file of the corresponding candidate calculation graph into the model training cache according to each hash value.

[0140] Optionally, the device further includes:

[0141] The cache query module is used to query the model training cache according to the target hash value of the currently issued calculation graph, obtain the executable file of the target candidate calculation graph corresponding to the target hash value, and perform model training.

[0142] The model training device provided in the embodiment of the present invention can execute the model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0143] Embodiment 4

[0144] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0145] like Fig.10As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0146] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0147] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for model training.

[0148] In some embodiments, the method of model training may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method of model training described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method of model training in any other appropriate manner (e.g., by means of firmware).

[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0151] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0154] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0155] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0156] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: include: Generate an initial computational graph based on the training script for model training; When the computation graph is issued, a memory static analysis is performed on the initial computation graph to obtain a computation graph memory analysis result; According to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is segmented to obtain multiple candidate calculation graphs; Compile each candidate computational graph to generate an executable file for model training.

2. The method according to claim 1, characterized in that Generate the initial computational graph based on the training script for model training, including: Determine when to generate the computational graph based on script execution operations and / or user instructions during model training; When the computation graph is generated, an initial computation graph is generated according to a training script for model training.

3. The method according to claim 1, characterized in that When the computation graph is issued, a static memory analysis is performed on the initial computation graph to obtain the computation graph memory analysis result, including: Topologically sorting the initial computation graph to obtain an operator execution sequence; Determining the memory size occupied by each computing node according to operator input information and operator output information in the operator execution sequence; Determining whether the memory statically allocated to the target computing node can be reclaimed according to the reference relationship between the computing nodes in the operator execution sequence; When performing a memory static analysis on a subsequent computing node of the target computing node, allocating memory to the subsequent computing node according to the recovered memory; Determine the computational graph memory analysis results based on the static memory allocation results of each computing node and the corresponding memory size.

4. The method according to claim 3, characterized in that Determining whether the memory statically allocated to the target computing node can be reclaimed according to the reference relationship between the computing nodes in the operator execution sequence includes: According to the operator execution sequence, mark the reference count of each computing node referenced by other computing nodes, and adjust the reference count according to the static analysis of the memory of each computing node; When the reference count of the target computing node is zero, the memory occupied by the target computing node is reclaimed.

5. The method according to claim 1, characterized in that According to the calculation graph memory analysis result and the remaining running memory of the model training executor, the initial calculation graph is segmented to obtain multiple candidate calculation graphs, including: The initial computation graph is segmented by a binary method to obtain a current candidate computation graph; and a memory static analysis is performed on the current candidate computation graph; When the memory analysis result of the current candidate computation graph is greater than the remaining running memory of the model training executor, the current candidate computation graph is segmented using the binary method to obtain the next candidate computation graph; Update the next candidate computation graph to the current candidate computation graph, and return to the memory static analysis step for the current candidate computation graph until the memory analysis result of the current candidate computation graph is less than or equal to the remaining running memory of the model training executor.

6. The method according to claim 1, characterized in that After compiling each candidate computational graph and generating an executable file for model training, it also includes: Determine the hash value of each candidate computation graph according to the node information of each computation node in each candidate computation graph; According to each of the hash values, the executable file of the corresponding candidate calculation graph is cached in the cache of model training.

7. The method according to claim 6, characterized in that Also includes: The model training cache is queried according to the target hash value of the currently issued calculation graph, the executable file of the target candidate calculation graph corresponding to the target hash value is obtained, and the model training is performed.

8. A device for model training, characterized in that: include: The initial calculation graph generation module is used to generate the initial calculation graph according to the training script of the model training; A memory static analysis module is used to perform memory static analysis on the initial calculation graph when the calculation graph is issued, and obtain the calculation graph memory analysis result; A computation graph segmentation module, used to segment the initial computation graph according to the computation graph memory analysis result and the remaining running memory of the model training executor to obtain multiple candidate computation graphs; The computational graph compilation module is used to compile each candidate computational graph and generate an executable file for model training.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model training method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model training method described in any one of claims 1 to 7 when executed.

Images