Optimization method and framework compiler of deep learning model framework compiler

Abstract

This invention discloses an optimization method for a deep learning model framework compiler and the framework compiler itself. The method includes: the framework compiler acquiring deep learning model information and hardware resource information; the framework compiler determining a distributed training strategy based on the deep learning model information, hardware resource information, and a preset strategy configuration algorithm; the framework compiler reconstructing the deep learning model and generating training code based on the distributed training strategy; and the framework compiler combining the generated training code with preset scheduling commands based on the hardware resource information and the distributed training strategy to schedule the deployment and execution of the training code within the computing resources.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an optimization method and a framework compiler for deep learning model frameworks. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Deep learning workloads need to be deployed on specific hardware to fully realize their potential. Currently, developers in the deep learning field have designed a variety of frameworks, such as Tensorflow, PyTorch, and PaddlePaddle, to complete the training and inference tasks of deep learning models. At the same time, hardware manufacturers have also launched a variety of hardware backends, such as GPUs and FPGAs, to accelerate the training and inference speed of deep learning models.

[0004] There is a need for a bridge between a large number of different deep learning frameworks and an increasing number of hardware architectures. As a complete optimization toolchain, the deep learning framework compiler provides an end-to-end solution for deploying deep learning workloads of different frameworks to various hardware devices.

[0005] The inventors discovered that the existing technology has a technical problem: for different deep learning frameworks, existing compilers have few parallel computing strategies to choose from (generally only supporting data parallelism and model parallelism), which cannot generate distributed training code that meets the needs of multiple deep learning frameworks and cannot fully utilize the storage and computing performance of computing resources. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an optimization method and framework compiler for deep learning model frameworks; through the hyperparameters and directives designed in this invention, convenient configuration of parallel strategies and automatic generation of distributed training code are achieved.

[0007] On the one hand, it provides optimization methods for deep learning model framework compilers;

[0008] Optimization methods for deep learning model framework compilers include:

[0009] The framework compiler obtains information about deep learning models and hardware resources.

[0010] The framework compiler determines the distributed training strategy based on deep learning model information, hardware resource information, and preset strategy configuration algorithms.

[0011] The framework compiler reconstructs the deep learning model and generates training code based on the distributed training strategy;

[0012] The framework compiler combines the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, and schedules the training code to be deployed and run within computing resources.

[0013] On the other hand, a framework compiler is provided;

[0014] Deep learning model framework compilers, including:

[0015] The hyperparameter input module is configured to acquire deep learning model information and hardware resource information.

[0016] The distributed training strategy configuration module is configured such that the framework compiler determines the distributed training strategy based on the deep learning model information, hardware resource information, and the preset strategy configuration algorithm.

[0017] The distributed training code generation module is configured to: reconstruct the deep learning model and generate training code according to the distributed training strategy;

[0018] The code execution module is configured to combine the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, and schedule the training code to be deployed and run within computing resources.

[0019] The above technical solution has the following advantages or beneficial effects:

[0020] The compiler selects and configures parallel computing strategies to generate distributed training code that meets storage and computing requirements, fully leveraging the storage and computing performance of computing resources.

[0021] A technique for agile access to deep learning models in multi-machine, multi-GPU environments is proposed, providing the ability to integrate models across deep learning frameworks and shortening the development and training cycle of large models. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a system flowchart of Example 1;

[0024] Figure 2 This is a flowchart of the distributed training strategy selection process in Example 1;

[0025] Figure 3 Flowchart for determining the distributed training strategy in Example 1;

[0026] Figure 4 This is the distributed training code generation process for Example 1;

[0027] Figure 5 The scheduling and training code flow is shown in Example 1;

[0028] Figure 6 This is a system architecture diagram for Example 1;

[0029] Figure 7 This illustrates the training acceleration in Example 1.

[0030] Figure 8 This is a schematic diagram illustrating the Tensorflow framework support in Example 1;

[0031] Figure 9 This is a schematic diagram illustrating the PyTorch framework support in Example 1. Detailed Implementation

[0032] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] In recent years, deep learning technology, aimed at optimizing model inference and prediction performance, has gradually matured and has been widely applied in industries such as medicine, industry, manufacturing, power, and transportation. With the advancement of big data technology, researchers have begun to focus more on deep learning models with larger training datasets and model parameters, hoping to achieve better inference and prediction results. Especially with the rise of large language models such as ChatGPT, the academic community has largely reached a consensus on the relationship between model size and model performance.

[0034] As the size of models continues to increase and the amount of training data grows, using a single GPU resource for deep learning model training can no longer meet the storage and computational requirements of parameter scale and data throughput. Parallel computing strategies and distributed GPU resources are needed to complete the training of large models.

[0035] Implementing distributed training requires decomposing the training data and model of a single-machine, single-GPU model and designing pipeline strategies to reduce synchronization latency. However, current single-machine, single-GPU deep learning models across various fields are implemented using different deep learning frameworks. Mainstream deep learning frameworks in my country include PyTorch, TensorFlow, and PaddlePaddle, each with its own implementation methods and configuration procedures for distributed training, leading to difficulties in implementing distributed training. Furthermore, common parallel computing strategies in distributed training include data parallelism, model parallelism, pipeline parallelism, and operator-level parallelism; how to rationally select and configure these strategies is also a common and complex problem.

[0036] To address these issues, researchers have explored automatic parallel strategy selection methods, generally categorized into fully automatic and semi-automatic parallelism. Semi-automatic parallelism requires the user to specify the partitioning methods for tensors and operators; fully automatic parallelism, on the other hand, delegates the partitioning process entirely to the framework for adaptive selection. However, for the problem of numerous deep learning models and the difficulty of implementing distributed training, a satisfactory solution remains elusive.

[0037] Example 1

[0038] This embodiment provides an optimization method for a deep learning model framework compiler;

[0039] like Figure 1 As shown, the optimization methods for the deep learning model framework compiler include:

[0040] S101: The framework compiler obtains deep learning model information and hardware resource information;

[0041] S102: The framework compiler determines the distributed training strategy based on the deep learning model information, hardware resource information, and preset strategy configuration algorithm.

[0042] S103: The framework compiler reconstructs the deep learning model and generates training code based on the distributed training strategy;

[0043] S104: The framework compiler combines the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, and schedules the training code to be deployed and run within computing resources.

[0044] Further, in S101: the framework compiler acquires deep learning model information and hardware resource information; wherein,

[0045] The deep learning model information includes: the deep learning model network structure, the deep learning model framework, the default parallel strategy, and the code execution phase;

[0046] The hardware resource information includes: the number of available GPUs, the master node address, the master node port number, the worker node address, and the worker node port number.

[0047] For example, the deep learning model includes, but is not limited to, feedforward neural networks, radial basis function neural networks, convolutional neural networks, recurrent convolutional neural networks, generative adversarial neural networks, long short-term memory neural networks, gated neural networks, etc.

[0048] Further, in S102: the framework compiler determines a distributed training strategy based on deep learning model information, hardware resource information, and a preset strategy configuration algorithm; wherein the distributed training strategy includes: data parallelism, model parallelism, distributed data parallelism, pipeline parallelism, and optimizer parallelism.

[0049] Furthermore, such as Figure 2 As shown, in step S102: the framework compiler determines the distributed training strategy based on deep learning model information, hardware resource information, and a preset strategy configuration algorithm. The determination steps include:

[0050] S102-1: Determine the set of distributed training strategies based on the deep learning model framework information;

[0051] S102-2: Calculate storage overhead and computational overhead based on the deep learning model structure information;

[0052] S102-3: Determine storage capacity and computing power based on hardware resource information;

[0053] S102-4: Determine the distributed training strategy based on storage overhead, computational overhead, storage capacity, computational capability, and the set of distributed training strategies.

[0054] Further, S102-1: Determining a set of distributed training strategies based on the deep learning model framework information, specifically including:

[0055] The PyTorch framework includes: data parallelism, model parallelism, distributed data parallelism, pipelined parallelism, and optimizer parallelism.

[0056] The Tensorflow framework includes: data parallelism, model parallelism, multi-machine data parallelism, and pipelined parallelism.

[0057] Further, S102-2: Calculating storage overhead and computational overhead based on the deep learning model structure information, specifically including:

[0058] The formula for calculating storage overhead M is:

[0059]

[0060] Where n is the number of layers in the neural network, P i V represents the number of parameters in the i-th layer of the neural network. i O represents the number of intermediate variables in the i-th layer of the neural network. i This represents the number of optimizer variables in the i-th layer of the neural network;

[0061] The formula for calculating the computational cost C is:

[0062]

[0063] Where n is the number of layers in the neural network, P i G represents the number of parameters in the i-th layer of the neural network. i This represents the number of gradient variables in the i-th layer of the neural network, and Comm is the additional communication overhead.

[0064] Further, step S102-3: determining the storage capacity and computing power based on hardware resource information, specifically includes:

[0065] The formula for calculating storage capacity M′ is:

[0066]

[0067] Where p is the number of available compute nodes, q is the number of GPUs in the i-th compute node, and m ij This represents the storage capacity of the j-th GPU in the i-th computing node;

[0068] The formula for calculating computational ability C′ is:

[0069]

[0070] Where p is the number of available compute nodes, q is the number of GPUs in the i-th compute node, and f ij sp represents the clock speed of the j-th GPU on the i-th compute node. ij This represents the number of stream processors in the j-th GPU of the i-th compute node.

[0071] Further, S102-4: Based on storage overhead, computational overhead, storage capacity, computational capability, and the set of distributed training strategies, a distributed training strategy is determined, specifically including:

[0072] Data parallelism, model parallelism, distributed data parallelism, pipeline parallelism, optimizer parallelism, and any combination thereof.

[0073] Data Parallel Strategy: In the data parallel strategy, the training task is split across multiple processes (GPU devices). Each process maintains the same model parameters and the same computational task, but processes different training data.

[0074] Model parallel strategy: In the model parallel strategy, a single model is split across multiple GPUs according to the neural network structure, and they work together to complete the training task of the same model and the same batch of data.

[0075] Distributed data parallel strategy: In addition to completing the data parallel strategy, it also adds communication and management functions across computing nodes.

[0076] Pipeline parallelism strategy: (After using model parallelism and data parallelism strategies, a large amount of data communication will generate a lot of additional waiting time between GPU devices.) The training data is divided into smaller micro-batches. When the micro-batch of data arrives at the current GPU, the previous micro-batch of data has been calculated and sent to the next GPU, thereby utilizing computation to hide communication and waiting time.

[0077] Parallel optimizer strategy: In a data-parallel device, the parameter, gradient, and optimizer state data (the data during model training includes three parts: parameters, gradient, and optimizer state) are sharded. Then, only one shard of parameters, gradient, and optimizer state is stored in each data-parallel device, thereby reducing storage overhead.

[0078] Furthermore, such as Figure 3 As shown, step S102-4: Based on storage overhead, computational overhead, storage capacity, computational capability, and the set of distributed training strategies, a distributed training strategy is determined, specifically including:

[0079] according to Allocate parallel devices for the model, the numerator M represents the model storage overhead, and the denominator M ′ Indicates storage capacity. This indicates that the calculation result is rounded down, and the result is the number of parallel devices in the model, i.e., the number of parallel devices within the group, denoted as G. within ;

[0080] like Then enable the optimizer parallel strategy;

[0081] If p > 1, then use the distributed data parallel strategy; otherwise, use the data parallel strategy.

[0082] according to Distribute data in parallel devices, molecules G represents the total number of GPUs on each compute node. within The number of parallel devices within a group is calculated to yield the number of data parallel devices, which is the number of parallel devices between groups, denoted as G. between ;

[0083] Obviously, The total number of computing devices is the product of the number of parallel devices within a group and the number of parallel devices between groups.

[0084] like Then a pipelined parallel strategy is used.

[0085] Optimizer parallel strategies and pipeline parallel strategies do not require additional allocation of parallel devices.

[0086] Furthermore, such as Figure 4As shown, S103: The framework compiler reconstructs the deep learning model and generates training code based on the distributed training strategy, specifically including:

[0087] S103-1: Determine the type of deep learning framework used by the deep learning model. If it is the deep learning framework PyTorch, proceed to S103-2; if it is the deep learning framework Tensorflow, proceed to S103-5.

[0088] S103-2: Based on the distributed training strategy, determine whether to divide the model code stage. If the distributed training strategy is model parallelism, pipeline parallelism, and optimizer parallelism, then divide the model code stage. If the distributed training strategy is data parallelism and distributed data parallelism, then divide the model code stage. If yes, proceed to S103-3; if no, proceed to S103-4.

[0089] S103-3: Generate distributed training code for each training stage according to the preset training stage code generation strategy; wherein, the preset training stage code generation strategy includes: package import, variable declaration, model splitting, data splitting, model encapsulation, data encapsulation or batch data input.

[0090] S103-4: Generate distributed training code according to a preset general code generation strategy; the preset general code generation strategy includes model encapsulation, hardware allocation, or data migration.

[0091] S103-5: According to the distributed training strategy, insert distributed training code line by line into the deep learning model. The distributed training code includes generating worker node IP addresses, port numbers, current node sequence numbers, encapsulation, and enabling the distributed training strategy.

[0092] Furthermore, such as Figure 5 As shown, S104: The framework compiler combines the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, scheduling the training code to be deployed and run within computing resources, specifically including:

[0093] S104-1: Determine the type of deep learning framework used by the deep learning model. If it is PyTorch, proceed to S104-2; if it is Tensorflow, proceed to S104-5.

[0094] S104-2: Generate PyTorch distributed training execution commands according to the preset command generation strategy. The preset command generation strategy includes training commands for single-machine multi-GPU environments and training commands for multi-machine multi-GPU environments.

[0095] A single-machine multi-GPU environment is a computing environment with multiple GPUs within an independent computing node.

[0096] A multi-machine, multi-GPU environment is a computing environment with multiple computing nodes and multiple GPUs within each node;

[0097] S104-3: Based on the hardware information, set the hyperparameters for the PyTorch execution command; the hyperparameters include: number of compute nodes, number of GPUs per node, current node number, master node address and / or master node port number;

[0098] S104-4: Based on the generated distributed training code, set the training code path for the PyTorch execution command, then proceed to S104-8;

[0099] S104-5: Generate Tensorflow distributed training execution commands according to the preset command generation strategy. The command format is python + generated code path + hyperparameters.

[0100] S104-6: Based on the deep learning model information, set the hyperparameters of the Tensorflow execution command. The hyperparameters of the Tensorflow execution command refer to the hyperparameters of the deep learning model itself.

[0101] S104-7: Based on the generated distributed training code, set the training code path for Tensorflow execution commands;

[0102] S104-8: Execute the distributed training process of the deep learning model.

[0103] This invention focuses on the distributed training requirements of large-scale deep learning models. It addresses the difficulties in implementing distributed training due to the variety of mainstream deep learning frameworks and the complexity of choosing parallel strategies. It designs and implements a distributed training code generation technology, which enables distributed parallel training of large-scale deep learning models in a multi-machine, multi-GPU environment through simple configuration, thereby accelerating the training process.

[0104] Example 2

[0105] This embodiment provides a framework compiler;

[0106] like Figure 6 As shown, the deep learning model framework compiler includes:

[0107] The hyperparameter input module is configured to acquire deep learning model information and hardware resource information.

[0108] The distributed training strategy configuration module is configured such that the framework compiler determines the distributed training strategy based on the deep learning model information, hardware resource information, and the preset strategy configuration algorithm.

[0109] The distributed training code generation module is configured to: reconstruct the deep learning model and generate training code according to the distributed training strategy;

[0110] The code execution module is configured to combine the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, and schedule the training code to be deployed and run within computing resources.

[0111] The deep learning model framework compiler of this invention, such as... Figure 2 As shown, it mainly includes three modules: hyperparameter input, automatic parallel strategy configuration, and distributed training code generation.

[0112] Furthermore, the hyperparameter input module, used to acquire and parse the most basic parameters for the model training process, belongs to the user interface layer. The input parameters supported by the hyperparameter input module and their descriptions are shown in Table 1. This invention acquires the hyperparameters shown in Table 1 and initializes the automatic parallel strategy configuration module and the distributed code implementation module based on the hyperparameter values.

[0113] Table 1. Parameter Description of Hyperparameter Input Module

[0114]

[0115] The distributed training strategy configuration module automatically decomposes the model structure, training data, and tensors to achieve model parallelism, data parallelism, pipeline parallelism, optimizer parallelism, and operator-level parallelism. This module comprises three components: a cost analyzer, a policy searcher, and a policy scheduler. The cost analyzer calculates the model's memory overhead, communication overhead, computational overhead, and the available resource limitations (memory constraints, computing power, and network bandwidth). The policy searcher searches for and combines available parallel strategies and calculates the optimized structure based on those strategies. The policy scheduler distributes the decomposed model structure, training data, and tensors to various computing devices.

[0116] The distributed training code generation module is used to generate distributed training code, modify the original model, and execute model training tasks based on the configuration information of the directives. The distributed training code generation module mainly includes four components: training control, model translation, parallel strategy support, and training driver.

[0117] The training control component is used to call the hyperparameter input module, the automatic parallel strategy configuration module, and the distributed training code generation module, and to control their calling process;

[0118] The model translation component is used to parse the guidance statements and translate the original model code into automatically generated distributed training code.

[0119] The parallel strategy support component is used to call the strategy search results of the automatic parallel strategy configuration module and further modify the distributed training code;

[0120] Training-driven components are used to execute the training process.

[0121] The code generation process for this module is as follows: input the original model code and hyperparameters; parse the hyperparameters and set shared variables; determine the pre-selected distributed training strategy; determine the current code execution stage; according to the execution stage, import the generated package, declare variables, split the model, split the data, encapsulate the model, encapsulate the data, or input the code in batches; add the generated code to the code execution sequence; reconstruct the original model and output the new model code.

[0122] This invention can automatically convert training code for single-machine, single-card models implemented in PyTorch or Tensorflow deep learning frameworks into runnable distributed training code using limited hyperparameters and guidance statements, thereby achieving a convenient distributed training process.

[0123] The modifications made to the original single-machine, single-card model by this invention are shown in Table 2. As can be seen from Table 2, only a very small number of hyperparameters need to be configured and a very small number of guidance statements need to be added to achieve the automatic generation of distributed training code and resource configuration, demonstrating that this invention can effectively simplify the implementation process of distributed training across multiple frameworks.

[0124] Table 2. Number of Hyperparameters and Guidance Statement Configurations

[0125]

[0126] After using the technology of this invention to automatically generate distributed training code, a good acceleration effect can be achieved in the distributed training process of deep learning models, such as... Figure 7 As shown. Figure 7 The statistics show the training time using 4 GPUs, where train represents the training time on a single machine and a single GPU. Other distributed training strategies achieved a certain degree of speedup compared to this, with DDP achieving a speedup of 5.41 times, FSDP achieving a speedup of 5.26 times, and MP achieving a speedup of 1.42 times. This demonstrates that the distributed training process implemented in this invention is effective. Figure 8 This is a schematic diagram illustrating the Tensorflow framework support in Example 1; Figure 9 This is a schematic diagram illustrating the PyTorch framework support in Example 1.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. Optimization methods for deep learning model framework compilers, characterized by: include: The framework compiler obtains information about deep learning models and hardware resources. The framework compiler determines the distributed training strategy based on deep learning model information, hardware resource information, and a pre-defined strategy configuration algorithm; the determination steps include: Based on the deep learning model framework information, determine the set of distributed training strategies; Based on the deep learning model structure information, calculate the storage overhead and computational overhead; Based on hardware resource information, determine storage capacity and computing power; Based on storage overhead, computational overhead, storage capacity, computational capability, and a set of distributed training strategies, a distributed training strategy is determined; specifically, this includes: according to Allocation model parallel devices, molecules The denominator represents the model storage overhead. Indicates storage capacity. This indicates that the calculation result is rounded down. The result is the number of parallel devices in the model, which is also the number of parallel devices within the group, denoted as . ; like If so, then the optimizer parallel strategy is enabled; like If the condition is met, a distributed data parallelism strategy is used; otherwise, a data parallelism strategy is used. according to Distribute data in parallel devices, molecules This represents the total number of GPUs on each compute node. The number of parallel devices within a group is calculated as the number of data parallel devices. The number of data parallel devices is also the number of parallel devices between groups, denoted as... ; = The product of the number of parallel devices within a group and the number of parallel devices between groups is the total number of computing devices. like If so, a pipelined parallel strategy is used; The framework compiler reconstructs the deep learning model and generates training code based on a distributed training strategy; specifically, this includes: Determine the type of deep learning framework used by the deep learning model. If it is PyTorch, determine whether to divide the model code into stages based on the distributed training strategy. If so, generate distributed training code for each training stage according to the preset training stage code generation strategy. The training stage includes: package import, variable declaration, model splitting, data splitting, model encapsulation, data encapsulation, or batch data input. If not, generate distributed training code according to the preset general code generation strategy. The distributed training code includes model encapsulation, hardware allocation, or data migration. If it is the deep learning framework Tensorflow, then according to the distributed training strategy, the distributed training code is inserted line by line. The distributed training code includes generating worker node information, encapsulating and enabling the distributed training strategy. The framework compiler combines the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, and schedules the training code to be deployed and run within computing resources.

2. The optimization method for the deep learning model framework compiler as described in claim 1, characterized in that, The framework compiler obtains deep learning model information and hardware resource information; among them, The deep learning model information includes: the deep learning model network structure, the deep learning model framework, the default parallel strategy, and the code execution phase; The hardware resource information includes: the number of available GPUs, the master node address, the master node port number, the worker node address, and the worker node port number. The framework compiler determines the distributed training strategy based on deep learning model information, hardware resource information, and preset strategy configuration algorithms; wherein the distributed training strategy includes: data parallelism, model parallelism, distributed data parallelism, pipeline parallelism, and optimizer parallelism.

3. The optimization method for the deep learning model framework compiler as described in claim 1, characterized in that, Based on the deep learning model framework information, a set of distributed training strategies is determined, specifically including: PyTorch framework, including: data parallelism, model parallelism, distributed data parallelism, pipeline parallelism, and optimizer parallelism; Tensorflow framework, including: data parallelism, model parallelism, multi-machine data parallelism, and pipeline parallelism.

4. The optimization method for the deep learning model framework compiler as described in claim 1, characterized in that, Based on the deep learning model structure information, calculate the storage overhead and computational overhead, specifically including: The formula for calculating storage overhead is: ， in, It refers to the number of layers in a neural network. This represents the number of parameters in the i-th layer of the neural network. This represents the number of intermediate variables in the i-th layer of the neural network. This represents the number of optimizer variables in the i-th layer of the neural network; The formula for calculating the cost is as follows: ， in, It refers to the number of layers in a neural network. This represents the number of parameters in the i-th layer of the neural network. This represents the number of gradient variables in the i-th layer of the neural network. It is an additional communication overhead.

5. The optimization method for the deep learning model framework compiler as described in claim 1, characterized in that, Based on hardware resource information, determine storage capacity and computing power, specifically including: Storage capacity The calculation formula is: Where p is the number of available compute nodes, and q is the number of GPUs in the i-th compute node. This represents the storage capacity of the j-th GPU in the i-th computing node; computing power The calculation formula is: Where p is the number of available compute nodes, and q is the number of GPUs in the i-th compute node. This represents the clock speed of the j-th GPU on the i-th compute node. This represents the number of stream processors in the j-th GPU of the i-th compute node.

6. The optimization method for the deep learning model framework compiler as described in claim 1, characterized in that, The framework compiler combines the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, scheduling the training code to be deployed and run within computing resources. Specifically, this includes: The system determines the type of deep learning framework used by the deep learning model. If it is PyTorch, it generates PyTorch distributed training execution commands according to a preset command generation strategy, which includes single-machine multi-GPU environment training commands and multi-machine multi-GPU environment training commands. Based on the hardware information, it sets the hyperparameters of the PyTorch execution commands, including the number of compute nodes, the number of GPUs on each node, the current node index, the master node address, and / or the master node port number. Based on the generated distributed training code, it sets the training code path for the PyTorch execution commands and executes the distributed training process of the deep learning model. If it is the deep learning framework Tensorflow, then according to the preset command generation strategy, Tensorflow distributed training execution commands are generated; according to the deep learning model information, the hyperparameters of the Tensorflow execution commands are set. The hyperparameters of the Tensorflow execution commands refer to the hyperparameters of the deep learning model itself; according to the generated distributed training code, the training code path of the Tensorflow execution commands is set; and the distributed training process of the deep learning model is executed.

7. A deep learning model framework compiler, characterized by: The optimization method of the deep learning model framework compiler as described in any one of claims 1-6 includes: The hyperparameter input module is configured to acquire deep learning model information and hardware resource information. The distributed training strategy configuration module is configured as follows: the framework compiler determines the distributed training strategy based on deep learning model information, hardware resource information, and a preset strategy configuration algorithm. The determination steps include: determining a set of distributed training strategies based on the deep learning model framework information; calculating storage and computational overhead based on the deep learning model structure information; determining storage capacity and computational capability based on hardware resource information; and determining the distributed training strategy based on storage overhead, computational overhead, storage capacity, computational capability, and the set of distributed training strategies. Specifically, this includes: according to... Allocation model parallel devices, molecules The denominator represents the model storage overhead. Indicates storage capacity. This indicates that the calculation result is rounded down. The result is the number of parallel devices in the model, which is also the number of parallel devices within the group, denoted as . ;like Then enable the optimizer parallel strategy; if If the condition is met, a distributed data parallelism strategy is used; otherwise, a data parallelism strategy is used. Distribute data in parallel devices, molecules This represents the total number of GPUs on each compute node. The number of parallel devices within a group is calculated as the number of data parallel devices. The number of data parallel devices is also the number of parallel devices between groups, denoted as... ; = The product of the number of parallel devices within a group and the number of parallel devices between groups is the total number of computing devices; if If so, a pipelined parallel strategy is used; The distributed training code generation module is configured to: reconstruct the deep learning model and generate training code according to a distributed training strategy; specifically, it includes: determining the type of deep learning framework used by the deep learning model; if it is the PyTorch deep learning framework, then determining whether to divide the model code into stages according to the distributed training strategy; if so, then generating distributed training code for each training stage according to a preset training stage code generation strategy; wherein the training stage includes: package import, variable declaration, model splitting, data splitting, model encapsulation, data encapsulation or batch data input; if not, generating distributed training code according to a preset general code generation strategy; the distributed training code includes model encapsulation, hardware allocation or data migration; if it is the Tensorflow deep learning framework, then inserting distributed training code line by line according to the distributed training strategy; the distributed training code includes generating worker node information, encapsulation and enabling the distributed training strategy; The code execution module is configured to combine the generated training code with preset scheduling commands based on hardware resource information and distributed training strategies, and schedule the training code to be deployed and run within computing resources.

Images