Large model hybrid parallel unloading strategy optimization method and system in video memory resource limited scene

Abstract

The invention discloses a method and a system for optimizing a hybrid parallel unloading strategy of a large model in a video memory resource limited scene, aiming at a hardware environment with limited video memory resources and combining the characteristics of application unloading and hybrid parallel training in a large model training process based on Transform, the training task of the model is efficiently matched with the limited video memory resources, and the optimization of the hybrid parallel unloading strategy of the large model in the video memory resource limited scene is realized. A model is divided into an inter-op stage and an inter-op stage, strategy search is carried out in a layered mode, and local optimal solutions are obtained on the premise that the search space is greatly reduced. By designing a strategy conversion algorithm, the searched hybrid parallel unloading strategy is converted into parallel strategy configuration which can be executed by a large model parallel training framework, and the training effect can be evaluated and verified. According to the method, finite video memory resources can be fully utilized, the influence of video memory bottlenecks on large model training is effectively relieved, and meanwhile, the model training efficiency is greatly improved through flexible parallel calculation.

Description

Technical Field

[0001] This invention belongs to the field of parallelization acceleration in computer system architecture, and specifically relates to an optimization method and system for large-scale hybrid parallel offloading strategy in scenarios with limited video memory resources. Background Technology

[0002] Since the emergence of Transformer-based deep learning models in 2017, the size of deep learning models has grown exponentially, with thousands of models emerging worldwide, leading to fierce competition. Large models have now exceeded trillions of parameters, thanks to the significant improvement in model accuracy resulting from the increased number of parameters. For example, scaling large language models like GPT to hundreds of billions of parameters and training them on larger datasets can significantly improve training metrics. However, the feasibility of training large models is severely limited by the memory capacity of hardware computing devices. The economic and time costs of training large models are rising daily. To complete the training of large models, it is necessary to deploy the training task on large distributed computing clusters, combined with advanced memory optimization techniques. Deploying distributed training clusters on large-scale heterogeneous hardware devices requires extensive engineering work on model structure and hardware environment. This work can be broadly divided into two directions: one is exploring complex combinations of parallel methods during training. These methods are roughly divided into manual search and automatic search. Manual search has poor flexibility, relies on expert experience, and may not find the optimal strategy, while automatic search faces a high-dimensional search space and has certain technical difficulties. Each has its advantages and disadvantages. Secondly, it makes full use of the storage capacity of heterogeneous resources to optimize the GPU memory of computing nodes. The parameters, gradients, and optimizer states of traditional distributed training models are stored in GPU memory, which consumes too much GPU memory resources. This can be achieved by offloading some data and low-computing-power-requirement calculations to the CPU, thereby making efficient use of GPU memory resources.

[0003] Therefore, optimizing and combining the work in these two directions has become a major challenge for distributed parallel training of large models. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for optimizing hybrid parallel offloading strategies for large models in scenarios with limited GPU memory resources. By dividing the model structure into two layers, intra-op and inter-op, and performing hierarchical search, the search space for offloading and parallel strategies is reduced, improving search efficiency. Furthermore, a strategy transformation algorithm is used to evaluate and verify the strategies. This solves the problems of mismatch between GPU memory resources and model training scale, and low training efficiency for large models.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: An optimization method for hybrid parallel unloading strategy of large models in scenarios with limited video memory resources includes the following steps: Step 1: Obtain the large model architecture, get the HLO computation graph, and obtain hardware computing power device information such as GPU memory capacity and bandwidth; Step 2: Based on the obtained HLO computation graph and hardware computing power device information, construct the communication, computation and memory usage costs, and divide the large model into two layers, intra-op and inter-op, in the pipeline parallel stage to search for the optimal hybrid parallel offloading strategy. Step 3: Use the strategy transformation algorithm to transform the optimal hybrid parallel offloading strategy into a parallel configuration that can be executed by the parallel training framework, perform training and record training data and metrics, evaluate the training efficiency of large models and the amount of GPU memory used during training, and verify the effect.

[0006] A further improvement of this invention is that, in step 1, the large model architecture is obtained, the HLO computation graph is obtained, and hardware computing power device information such as GPU memory capacity and bandwidth is obtained, including: First, the basic architecture of the large model is obtained using a Python script. Its computation is then decomposed into repeatedly stacked operator blocks. HLO computation graph simulation is performed on all operators within each operator block. For each operator in the computation graph, the HLO computation graph is simulated using a class, and the simulation content is divided into three parts: First, the shape of the operator input-output tensor is simulated. This part ensures the correct construction of the computation graph and the correctness of the strategy simulation. Second, the operator partitioning strategy is generated based on the hardware information of the computing device, and the corresponding execution overhead is simulated, including communication overhead, computation overhead, memory overhead, and tensor repartitioning overhead, which are used for subsequent steps to search for the optimal parallel strategy. Finally, based on the computation graph and execution overhead obtained in the first two steps, the optimal hybrid parallel strategy is searched.

[0007] A further improvement of this invention is that, in step 2, based on the obtained HLO computation graph and hardware computing power information, the communication, computation, and memory usage costs are constructed. The large model is divided into intra-op and inter-op layers in a pipelined parallel phase, and the optimal hybrid parallel offloading strategy is searched for separately, including: Step 2.1: Based on the obtained HLO computation graph, hardware computing power device information and global communication configuration, perform in-stage strategy search for the pipeline stages that the model may be divided into, specifically using the Integer Linear Programming (ILP) algorithm. Step 2.2: Based on the obtained strategies within each stage and combined with the view of the hardware device, explore the partitioning of each stage on the device subgrid. Find the optimal pipeline parallel strategy, i.e. the minimum training cost, through dynamic programming algorithm. Combine the hybrid parallel offloading strategy within each stage to output the optimal hybrid parallel offloading strategy.

[0008] A further improvement of this invention is that, in step 2.1, the search for parallel strategies within a stage uses an integer linear programming algorithm, and tensor partitioning is used to define the layout of tensors. For an N-dimensional tensor, if the i-th dimension is partitioned, it is represented by the symbol S; otherwise, it is represented by R to indicate the replication of the tensor. After defining the tensor axis for partitioning, the partitioned tensor axis is then defined on the device grid axis. The notation is reflected by adding a superscript to the tensor partitioning symbol. For the representation of model operators and data flow, the experimental platform uses the HLO computation graph representation method to simulate the dependencies between computation graph nodes, and defines the communication and computational overhead of each node operation, as well as the communication overhead caused by data resharding operations between operators. The optimization objective of the Intra-op layer is to select a tensor partitioning strategy for each node in the computation graph G = (V, E), execute it in parallel, and minimize the execution time of the entire computation graph. Specifically, this is to minimize the sum of the computation and communication costs of all nodes v∈V and the repartitioning communication costs of all edges e∈E, as shown in formula (1): (1) The first term in the formula represents the computation and communication overhead of the node, while the second term represents the communication cost of re-sharding. It is a node The algorithm selects the vector. It is a node The communication cost vector, It is a node The computational cost vector, From node To the node The resharding cost matrix It is the set of all nodes in the computation graph. It is the set of all edges in the computation graph.

[0009] A further improvement of this invention is that, in step 2.2, the inter-op layer receives the intra-stage hybrid parallel offloading strategy calculated by the intra-op layer, divides the computation graph into s stages between stages, each stage consists of a subset of corresponding operators, and assigns a sub-mesh of size ni×mi to each stage si. This mesh is sliced ​​from the logical topology graph of the hardware device, which is represented as a cluster mesh N×M. Then, for the overhead of each stage, the result returned by the upper ILP layer is used, which is expressed as ti =tintra(si, Mesh(ni,mi)). Assuming that there are B micro-batch in the pipeline parallelism, the total time cost of the computation graph is expressed as formula (2): (2) The total time cost includes two items: the first is the total cost of pipeline parallelizing all stages, and the second is the cost of the remaining B stages. 1. Micro-batch pipeline execution time; It minimizes the end-to-end pipeline execution latency. It is the first Each stage of the assembly line It is the first The device subgrid size allocated for each stage It is the first Execution delays at each stage This is the quantity of a micro-batch. This represents the total number of stages in the production line. For the exploration of minimizing the total delay, dynamic programming is used. For formula (2), DP first enumerates the second term tmax and minimizes the result of the first term for different tmax. Formula (3) is used to represent the minimum total cost when the operator ok is sliced ​​into s stages and placed on d devices: (3) in, Indicates from operation Divided into at the end Each stage, use Each device, each stage delay not exceeding The minimum total delay, From the first To the A subgraph consisting of operations. It is the shape of the equipment grid. This is the execution delay of the subgraph on the specified grid. This is the total number of operations; Where DP is initialized as F(0, K+1, 0; tmax) = 0, then the total delay is expressed as formula (4): (4) in, It is the overall shape of the cluster device mesh; After a hierarchical search through two stages, intra-op and inter-op, the strategy divides the model into n pipelined parallel stages, each of which is mapped to a sub-mesh of the hardware device mesh, and assigns a tensor partitioning parallel strategy to the intra-op layer.

[0010] A further improvement of this invention is that, in step 3, a strategy transformation algorithm is used to transform the optimal hybrid parallel offloading strategy into a parallel configuration executable by the parallel training framework, training is performed and training data and metrics are recorded, the training efficiency of large models and the amount of GPU memory used during training are evaluated, and the effect is verified, including: A policy transformation algorithm is employed to convert tensor partitioning and data offloading into parallel process groups and data offloading schemes suitable for the Megatron distributed parallel framework developed by NVIDIA. The algorithm first generates parallel process groups based on the logical topology of the hardware device and the hybrid parallel strategy, supporting tensor parallelism, pipelined parallelism, and data parallelism. By calculating the size and number of each process group and using a formula to decompose the global rank into the corresponding parallel group rank, the tensor partitioning strategy is mapped to the Megatron parallel process groups. After the policy transformation is completed, the generated parallel process groups and data offloading schemes are compared with existing schemes through practical testing within the Megatron framework. Evaluation metrics include training time, communication overhead, and memory utilization, thereby verifying the effectiveness of the policy search.

[0011] A system for optimizing hybrid parallel offloading strategies for large models in memory-constrained scenarios includes: The large model architecture acquisition unit acquires the large model architecture, obtains the HLO computation graph, and acquires hardware computing power device information such as GPU memory capacity and bandwidth. The large model is divided into units. Based on the obtained HLO computation graph and hardware computing power information, the communication, computation and memory usage costs are constructed. The large model is divided into two layers, intra-op and inter-op, in the pipeline parallel stage, and the optimal hybrid parallel offloading strategy is searched for separately. The validation unit is optimized by using a policy transformation algorithm to convert the optimal hybrid parallel offloading strategy into a parallel configuration executable by the parallel training framework. Training is then performed and training data and metrics are recorded to evaluate the training efficiency of large models and the amount of GPU memory used during training, and the results are verified.

[0012] A further improvement of this invention lies in that, in the large model architecture acquisition unit, the large model architecture is acquired, the HLO computation graph is obtained, and hardware computing power device information such as GPU memory capacity and bandwidth is acquired, including: First, the basic architecture of the large model is obtained using a Python script. Its computation is then decomposed into repeatedly stacked operator blocks. HLO computation graph simulation is performed on all operators within each operator block. For each operator in the computation graph, the HLO computation graph is simulated using a class, and the simulation content is divided into three parts: First, the shape of the operator input-output tensor is simulated. This part ensures the correct construction of the computation graph and the correctness of the strategy simulation. Second, the operator partitioning strategy is generated based on the hardware information of the computing device, and the corresponding execution overhead is simulated, including communication overhead, computation overhead, memory overhead, and tensor repartitioning overhead, which are used for subsequent steps to search for the optimal parallel strategy. Finally, based on the computation graph and execution overhead obtained in the first two steps, the optimal hybrid parallel strategy is searched.

[0013] A further improvement of this invention lies in that, within the large model partitioning unit, based on the obtained HLO computation graph and hardware computing power information, the communication, computation, and memory usage costs are constructed. The large model is then divided into intra-op and inter-op layers in a pipelined parallel phase, and the optimal hybrid parallel offloading strategy is searched for separately, including: Step 2.1: Based on the obtained HLO computation graph, hardware computing power device information and global communication configuration, perform in-stage strategy search for the pipeline stages that the model may be divided into, specifically using the Integer Linear Programming (ILP) algorithm. Step 2.2: Based on the obtained strategies within each stage and combined with the view of the hardware device, explore the partitioning of each stage on the device subgrid. Find the optimal pipeline parallel strategy, i.e. the minimum training cost, through dynamic programming algorithm. Combine the hybrid parallel offloading strategy within each stage to output the optimal hybrid parallel offloading strategy.

[0014] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the large-model hybrid parallel offloading strategy optimization method under the scenario of limited video memory resources.

[0015] Compared with the prior art, the present invention has at least the following beneficial technical effects: This invention provides a method and system for optimizing hybrid parallel offloading strategies for large models in scenarios with limited GPU memory. By dividing the model structure into two layers, intra-op and inter-op, and performing hierarchical search, the search space for offloading and parallel strategies is reduced, improving search efficiency. Furthermore, a strategy transformation algorithm is used to evaluate and validate the strategies. This solves the problems of mismatch between GPU memory resources and model training scale, and low training efficiency for large models. Attached Figure Description To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the method of the present invention.

[0017] Figure 2 This is a structural block diagram of the system of the present invention. Detailed Implementation

[0018] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0019] In the description of this invention, it should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0023] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] Example 1 The present invention provides an optimization method for hybrid parallel unloading strategy of large models in scenarios with limited video memory resources, comprising the following steps: Step 1: Obtain the large model architecture, get the HLO computation graph, and obtain hardware computing power device information such as GPU memory capacity and bandwidth; Step 2: Based on the obtained HLO computation graph and hardware computing power device information, construct the communication, computation and memory usage costs, and divide the large model into two layers, intra-op and inter-op, in the pipeline parallel stage to search for the optimal hybrid parallel offloading strategy. Step 3: Use the strategy transformation algorithm to transform the optimal hybrid parallel offloading strategy into a parallel configuration that can be executed by the parallel training framework, perform training and record training data and metrics, evaluate the training efficiency of large models and the amount of GPU memory used during training, and verify the effect.

[0025] In step 1 of this embodiment, the large model architecture is obtained, the HLO computation graph is obtained, and hardware computing power device information such as GPU memory capacity and bandwidth is obtained, including: First, the basic architecture of the large model is obtained using a Python script. Its computation is then decomposed into repeatedly stacked operator blocks. HLO computation graph simulation is performed on all operators within each operator block. For each operator in the computation graph, the HLO computation graph is simulated using a class, and the simulation content is divided into three parts: First, the shape of the operator input-output tensor is simulated. This part ensures the correct construction of the computation graph and the correctness of the strategy simulation. Second, the operator partitioning strategy is generated based on the hardware information of the computing device, and the corresponding execution overhead is simulated, including communication overhead, computation overhead, memory overhead, and tensor repartitioning overhead, which are used for subsequent steps to search for the optimal parallel strategy. Finally, based on the computation graph and execution overhead obtained in the first two steps, the optimal hybrid parallel strategy is searched.

[0026] In step 2 of this embodiment, based on the obtained HLO computation graph and hardware computing power information, the communication, computation, and memory usage costs are constructed. The large model is divided into intra-op and inter-op layers in a pipelined parallel phase, and the optimal hybrid parallel offloading strategy is searched for separately, including: Step 2.1: Based on the obtained HLO computation graph, hardware computing power device information and global communication configuration, perform in-stage strategy search for the pipeline stages that the model may be divided into, specifically using the Integer Linear Programming (ILP) algorithm. Step 2.2: Based on the obtained strategies within each stage and combined with the view of the hardware device, explore the partitioning of each stage on the device subgrid. Find the optimal pipeline parallel strategy, i.e. the minimum training cost, through dynamic programming algorithm. Combine the hybrid parallel offloading strategy within each stage to output the optimal hybrid parallel offloading strategy.

[0027] In step 2.1 of this embodiment, the search for parallel strategies within a stage uses an integer linear programming algorithm. Tensor partitioning is used to define the layout of tensors. For an N-dimensional tensor, if the i-th dimension is partitioned, it is represented by the symbol S; otherwise, it is represented by R to indicate the copying of the tensor. After defining the tensor axis for partitioning, the partitioned tensor axis is then defined on the device grid axis. The notation is reflected by adding a superscript to the tensor partition symbol. For the representation of model operators and data flow, the experimental platform uses the HLO computation graph representation method to simulate the dependencies between computation graph nodes and defines the communication and computational overhead of each node operation, as well as the communication overhead caused by data resharding operations between operators. The optimization objective of the Intra-op layer is to select a tensor partitioning strategy for each node in the computation graph G = (V, E), execute it in parallel, and minimize the execution time of the entire computation graph. Specifically, this is to minimize the sum of the computation and communication costs of all nodes v∈V and the repartitioning communication costs of all edges e∈E, as shown in formula (1): (1) The first term in the formula represents the computation and communication overhead of the node, while the second term represents the communication cost of re-sharding. It is a node The algorithm selects the vector. It is a node The communication cost vector, It is a node The computational cost vector, From node To the node The resharding cost matrix It is the set of all nodes in the computation graph. It is the set of all edges in the computation graph.

[0028] In step 2.2 of this embodiment, the inter-op layer receives the intra-stage hybrid parallel offloading strategy calculated by the intra-op layer, divides the computation graph into s stages between stages, each stage consists of a subset of corresponding operators, and assigns a sub-mesh of size ni×mi to each stage si. This mesh is sliced ​​from the logical topology graph of the hardware device. The logical topology graph is represented as a cluster mesh N×M. Then, for the overhead of each stage, using the result returned by the upper ILP layer, it is expressed as ti =tintra(si, Mesh(ni,mi)). Assuming that there are B micro-batch in the pipeline parallelism, the total time cost of the computation graph is expressed as formula (2): (2) The total time cost includes two items: the first is the total cost of pipeline parallelizing all stages, and the second is the cost of the remaining B stages. 1. Micro-batch pipeline execution time; It minimizes the end-to-end pipeline execution latency. It is the first Each stage of the assembly line It is the first The device subgrid size allocated for each stage It is the first Execution delays at each stage This is the quantity of a micro-batch. This represents the total number of stages in the production line. For the exploration of minimizing the total delay, dynamic programming is used. For formula (2), DP first enumerates the second term tmax and minimizes the result of the first term for different tmax. Formula (3) is used to represent the minimum total cost when the operator ok is sliced ​​into s stages and placed on d devices: (3) in, Indicates from operation Divided into at the end Each stage, use Each device, each stage delay not exceeding The minimum total delay, From the first To the A subgraph consisting of operations. It is the shape of the equipment grid. This is the execution delay of the subgraph on the specified grid. This is the total number of operations; Where DP is initialized as F(0, K+1, 0; tmax) = 0, then the total delay is expressed as formula (4): (4) in, It is the overall shape of the cluster device mesh; After a hierarchical search through two stages, intra-op and inter-op, the strategy divides the model into n pipelined parallel stages, each of which is mapped to a sub-mesh of the hardware device mesh, and assigns a tensor partitioning parallel strategy to the intra-op layer.

[0029] In step 3 of this embodiment, the optimal hybrid parallel offloading strategy is transformed into a parallel configuration executable by the parallel training framework using a strategy transformation algorithm. Training is then performed, and training data and metrics are recorded. The training efficiency of large models and the amount of GPU memory used during training are evaluated, and the effectiveness is verified, including: A policy transformation algorithm is employed to convert tensor partitioning and data offloading into parallel process groups and data offloading schemes suitable for the Megatron distributed parallel framework developed by NVIDIA. The algorithm first generates parallel process groups based on the logical topology of the hardware device and the hybrid parallel strategy, supporting tensor parallelism, pipelined parallelism, and data parallelism. By calculating the size and number of each process group and using a formula to decompose the global rank into the corresponding parallel group rank, the tensor partitioning strategy is mapped to the Megatron parallel process groups. After the policy transformation is completed, the generated parallel process groups and data offloading schemes are compared with existing schemes through practical testing within the Megatron framework. Evaluation metrics include training time, communication overhead, and memory utilization, thereby verifying the effectiveness of the policy search.

[0030] Example 2 This invention provides an optimization method for hybrid parallel offloading strategies of large models in scenarios with limited GPU memory. This method addresses hardware environments with limited GPU memory by combining the characteristics of offloading and hybrid parallel training in the training process of large Transformer-based models. It efficiently matches the model's training task with limited GPU memory resources by dividing the model into intra-op and inter-op stages and performing policy search layer by layer. This approach obtains local optima in each stage while significantly reducing the search space. A policy transformation algorithm is designed to convert the searched hybrid parallel offloading strategy into a parallel policy configuration executable by large model parallel training frameworks (such as Megatron), allowing for the evaluation and verification of training performance.

[0031] Please see Figure 1 The present invention provides an optimization method for hybrid parallel unloading strategy of large models in scenarios with limited video memory resources, comprising the following steps: Step 1: Extract the basic architecture of the large model using a Python script and decompose its computation process into repeatedly stacked operator blocks. For each operator block, model the operators using an HLO computation graph. Each operator is modeled using a specific Python class; for example, matrix multiplication uses the HloDot class. The simulation process mainly covers three aspects: first, simulating the shape of the operator input and output tensors to ensure the correct construction of the computation graph; second, generating operator partitioning strategies based on hardware computing power information, including tensor dimension partitioning and device mapping; and third, computational execution overhead, including communication, computation, memory, and tensor reharding overhead, to provide data support for subsequent solution steps.

[0032] Step 2.1 involves searching for an intra-stage parallel strategy using the Integer Linear Programming (ILP) algorithm and defining the tensor layout using a tensor partitioning specification. For an N-dimensional tensor, the i-th dimension being partitioned is represented by the symbol "S", while not partitioned is represented by "R" indicating tensor copying. For example, in the partitioning specification for a two-dimensional matrix, "SR" indicates partitioning by row, "RS" indicates partitioning by column, "SS" indicates partitioning by both row and column, and "RR" indicates no partitioning. After defining the partitioning, the tensor axes are mapped to the device mesh axes, indicated by adding superscripts to the symbols. For example, "S..." 0 "R" indicates that the tensor is split row-wise, with the first part copied on devices 0 and 1, and the second part copied on devices 2 and 3. When the output tensor of the previous operator does not conform to the input specification of the current operator, the tensor needs to be re-sharded. This may involve cross-device communication, such as changing "S". 0 Converting "R" to "RR" requires performing an all-gather operation along the 0th axis, resulting in additional communication overhead. This communication overhead is also included in the optimization objective of the integer linear programming algorithm.

[0033] The experimental platform uses an HLO computation graph to represent the process of model operators and data flow, simulates the dependencies between computation graph nodes, and defines the operation communication, computational overhead, and memory usage of each node.

[0034] In summary, the optimization objective of the Intra-op layer is to select a tensor partitioning strategy for each node (operator) in the computation graph G = (V, E), and execute it in parallel within the device subgrid to minimize the overall execution time of the computation graph. Specifically, this involves minimizing the sum of computation and communication costs on all nodes v∈V and repartitioning communication costs on all edges e∈E, as shown in Equation 1. (1) in, It is a node The algorithm selects vectors (one-hot). It is a node The communication cost vector, It is a node The computational cost vector, From node To the node The resharding cost matrix It is the set of all nodes in the computation graph. It is the set of all edges in the computation graph. The goal of this formula is to minimize the execution cost of the entire computation graph, including the computation and communication costs of all nodes, as well as the resharding costs incurred between nodes due to data layout transformations. Furthermore, this optimization objective is subject to memory usage limitations; that is, the amount of computational data generated during training cannot exceed the hardware's memory capacity.

[0035] Step 2.2: In the inter-op layer, the intra-stage hybrid parallel offloading strategy computed by the intra-op layer is received. Subsequently, the computation graph is divided into s stages, each consisting of a corresponding set of operators. For each stage s... i Allocate a size of n i ×m i The subgrids are obtained by partitioning the logical topology graph of the hardware device. The logical topology graph is represented as an N × M clustered grid. The cost of each stage is calculated using the integer linear programming (ILP) results returned by the intra-op layer, denoted as t. i =t intra (s i Mesh(n i ,m i )).

[0036] Assuming the pipeline runs in parallel with B micro-batches, the total time cost of the computation graph can be expressed as Equation 2: (2) in, It minimizes the end-to-end pipeline execution latency. It is the first Each stage of the assembly line It is the first The device subgrid size allocated for each stage It is the first Execution delays at each stage This is the quantity of a micro-batch. This is the total number of pipeline stages. The total latency consists of two parts: the first part is the total time for all stages to process the first microbatch, and the second part is the remaining latency. The processing time of a micro-batch at the slowest stage of the pipeline is limited by the slowest stage, and is reflected in the formula as t. j The value is the largest.

[0037] For minimizing the total delay, this method uses dynamic programming (DP). For Equation 2, DP first enumerates the second term t. max And for different t max Minimize the result of the first term, where t max This is the result obtained in step 2.1. We use Equation 3 to represent the operator o k Minimum total cost when s slices to s stages and place them on d devices: (3) in, Indicates from operation Divided into at the end Each stage, use Each device, each stage delay not exceeding The minimum total delay, From the first To the A subgraph consisting of operations. It is the shape of the equipment grid. This is the execution delay of the subgraph on the specified grid. This represents the total number of operations. The formula recursively tries different stage divisions and equipment allocation schemes to find the optimal pipeline configuration. DP is initialized as F(0, K+1, 0; t max If ) = 0, then the total delay can be expressed as Equation 4: (4) in, This represents the overall shape of the cluster device mesh, with the remaining symbols identical to those in formulas (2) and (3). This formula iterates through all possible number of stages. Find in satisfying The minimum total latency under constraints is added to the processing time of the remaining micro-batches in the pipeline. After a hierarchical search through two stages, intra-op and inter-op, the strategy divides the model into n pipelined parallel stages, each mapped to a sub-mesh of the hardware device mesh, and assigns a tensor partitioning parallel strategy for the intra-op layer.

[0038] Step 3: For the hybrid parallel offloading strategy obtained in Step 2, this method employs a strategy transformation algorithm to convert tensor partitioning and offloading data into parallel process groups and data offloading schemes suitable for the Megatron distributed parallel framework. Based on the logical topology of the hardware device and the hybrid parallel strategy (including tensor parallelism (TP), pipelined parallelism (PP), and data parallelism (DP), the algorithm generates parallel process groups and decomposes the global rank into the corresponding parallel group rank using a formula, thereby efficiently mapping the tensor partitioning strategy to the Megatron parallel process groups.

[0039] Simultaneously, the algorithm divides the tensor into multiple sub-tensors based on the tensor's dimension and parallel strategy, and distributes them across different devices, achieving data offloading through communication with the host. After completing the policy transformation, it is tested in the Megatron framework, comparing the generated parallel process groups and data offloading scheme with existing schemes. Evaluation metrics include training time, communication overhead, and memory utilization to verify the effectiveness of the policy search. This policy transformation algorithm not only transforms the theoretical hybrid parallel strategy into a practically executable distributed training scheme but also achieves an efficient training process within the Megatron framework.

[0040] After completing the above steps, the effectiveness evaluation of the hybrid parallel strategy is obtained. The automatic search strategy can then be improved and the search process optimized based on application requirements. This concludes the implementation method for optimizing the hybrid parallel offloading strategy for large models in scenarios with limited GPU memory resources.

[0041] Example 3 like Figure 2 As shown, the large-model hybrid parallel offloading strategy optimization system for memory-constrained scenarios provided by this invention includes: The large model architecture acquisition unit acquires the large model architecture, obtains the HLO computation graph, and acquires hardware computing power device information such as GPU memory capacity and bandwidth. The large model is divided into units. Based on the obtained HLO computation graph and hardware computing power information, the communication, computation and memory usage costs are constructed. The large model is divided into two layers, intra-op and inter-op, in the pipeline parallel stage, and the optimal hybrid parallel offloading strategy is searched for separately. The validation unit is optimized by using a policy transformation algorithm to convert the optimal hybrid parallel offloading strategy into a parallel configuration executable by the parallel training framework. Training is then performed and training data and metrics are recorded to evaluate the training efficiency of large models and the amount of GPU memory used during training, and the results are verified.

[0042] In the large model architecture acquisition unit of this embodiment, the large model architecture is acquired, the HLO computation graph is obtained, and hardware computing power device information such as GPU memory capacity and bandwidth is acquired, including: First, the basic architecture of the large model is obtained using a Python script. Its computation is then decomposed into repeatedly stacked operator blocks. HLO computation graph simulation is performed on all operators within each operator block. For each operator in the computation graph, the HLO computation graph is simulated using a class, and the simulation content is divided into three parts: First, the shape of the operator input-output tensor is simulated. This part ensures the correct construction of the computation graph and the correctness of the strategy simulation. Second, the operator partitioning strategy is generated based on the hardware information of the computing device, and the corresponding execution overhead is simulated, including communication overhead, computation overhead, memory overhead, and tensor repartitioning overhead, which are used for subsequent steps to search for the optimal parallel strategy. Finally, based on the computation graph and execution overhead obtained in the first two steps, the optimal hybrid parallel strategy is searched.

[0043] In the large model partitioning unit of this embodiment, based on the obtained HLO computation graph and hardware computing power information, the communication, computation, and memory usage costs are constructed. The large model is divided into intra-op and inter-op layers in a pipelined parallel phase, and the optimal hybrid parallel offloading strategy is searched for, including: Step 2.1: Based on the obtained HLO computation graph, hardware computing power device information and global communication configuration, perform in-stage strategy search for the pipeline stages that the model may be divided into, specifically using the Integer Linear Programming (ILP) algorithm. Step 2.2: Based on the obtained strategies within each stage and combined with the view of the hardware device, explore the partitioning of each stage on the device subgrid. Find the optimal pipeline parallel strategy, i.e. the minimum training cost, through dynamic programming algorithm. Combine the hybrid parallel offloading strategy within each stage to output the optimal hybrid parallel offloading strategy.

[0044] Example 4 The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the large-model hybrid parallel offloading strategy optimization method under the scenario of limited video memory resources.

[0045] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] This application is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0047] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0048] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0049] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0050] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for optimizing large model hybrid parallel offloading strategies in a graphics memory resource constrained scenario, characterized in that, The method comprises the following steps: Step 1, obtaining a large model architecture, obtaining an HLO calculation graph, and obtaining hardware computing device information of GPU memory capacity and bandwidth; Step 2, according to the obtained HLO calculation graph and hardware computing device information, constructing communication, calculation and memory occupation costs, and dividing the large model into intra-op and inter-op two layers in a pipeline parallel stage to obtain an optimal mixed parallel offloading strategy; Step 3, using a strategy conversion algorithm to convert the optimal mixed parallel offloading strategy into a parallel training framework executable parallel configuration, executing training and recording training data and indicators, evaluating the large model training efficiency and memory usage in the training process, and verifying the effect.

2. The method of claim 1, wherein the method is characterized in that, In step 1, the large model architecture is obtained, the HLO calculation graph is obtained, and the hardware computing device information of GPU memory capacity and bandwidth is obtained, including: First, the basic architecture of the large model is obtained through a python script, and the calculation is decomposed into a repeated stacked operator block, and HLO calculation graph simulation is performed on all operators in the operator block; for each operator in the calculation graph, an HLO calculation graph is used to simulate the content, which is divided into three parts: First, simulate the shape of the operator input and output tensor, which ensures the correctness of the calculation graph and the correctness of the strategy simulation; second, generate the splitting strategy of the operator according to the hardware information of the computing device and simulate the corresponding execution overhead, including communication overhead, calculation overhead, memory overhead and tensor re-sharding overhead, which is used for subsequent steps to search for the optimal parallel strategy; finally, according to the calculation graph and execution overhead obtained in the previous two steps, the optimal mixed parallel strategy is searched.

3. The method of claim 1, wherein, In step 2, according to the obtained HLO calculation graph and hardware computing device information, the communication, calculation and memory occupation costs are constructed, and the large model is divided into intra-op and inter-op two layers in a pipeline parallel stage to obtain an optimal mixed parallel offloading strategy, including: Step 2.1, according to the obtained HLO calculation graph, hardware computing device information and global communication configuration, the strategy search in the stage is performed on the possible pipeline stages of the model, which is realized by using integer linear programming ILP algorithm; Step 2.2, according to the obtained strategy in the stage, combining the view of the hardware device, exploring the division on each stage in the device subgrid, finding the optimal pipeline parallel strategy by dp dynamic programming algorithm, that is, the minimum training cost, combining the mixed parallel offloading strategy in each stage, outputting the optimal mixed parallel offloading strategy.

4. The method of claim 3, wherein, In step 2.1, for the search of the stage-in parallel strategy, an integer linear programming algorithm is used, and the tensor partitioning specification is used to define the layout of the tensor. For an N-dimensional tensor, if the ith dimension is partitioned, it is represented by the symbol S, otherwise, the replication of the tensor is represented by R. After defining the tensor axis for partitioning, the partitioned tensor axis is defined to the device grid axis. The practice is embodied in the symbol by adding a superscript to the symbol of tensor partitioning. For the representation of model operators and data flow, the experimental platform uses the HLO computation graph representation method to simulate the dependency relationship of the computation graph nodes, and defines the communication and computation overhead of each node operation, as well as the communication overhead caused by the data reshuffling operation between operators; The optimization goal of the intra-op layer is to select a tensor partitioning strategy for each node in the computation graph G = (V, E) and execute in parallel to minimize the execution time of the entire computation graph. The specific approach is to minimize the sum of the computation and communication costs of all nodes v∈V and the reshuffling communication costs of all edges e∈E, as shown in equation (1): （1） where the first term is the computation and communication cost of the node, and the second term represents the communication cost of resharding, is the algorithm selection vector of the node , is the communication cost vector of the node , is the computation cost vector of the node , is the resharding cost matrix from node to node , is the set of all nodes in the computation graph, is the set of all edges in the computation graph.

5. The method of claim 4, wherein, In step 2.2, the inter-op layer receives the stage-in mixed parallel offload strategy calculated by the intra-op layer, and divides the computation graph into s stages between stages, each stage consisting of a subset of corresponding operators. Each stage si is assigned a sub-grid with a size of ni×mi, which is sliced from the logical topology of the hardware device, represented as a cluster grid N×M. For the overhead of each stage, the results returned by the ILP of the previous layer are used, represented as ti = tintra(si, Mesh(ni,mi)). Assuming that there are B micro-batches in pipeline parallelism, the total time cost of the computation graph is represented as equation (2): （2） where the total time overhead includes two terms: the first term is the total overhead of pipelining all stages in parallel, and the second term is the overhead of the remaining B 1 micro-batch pipeline execution time; is the minimized end-to-end pipeline execution delay, is the execution delay of the first pipeline stage, is the execution delay of the first stage assigned device sub-grid size, is the execution delay of the first stage, is the number of micro-batches, is the total number of pipeline stages; For the exploration of minimizing the total delay, a dynamic programming method is used. For equation (2), DP first enumerates the second term tmax, and minimizes the result of the first term for different tmax. The minimum total overhead when slicing the operator ok into s stages and placing it on d devices is represented using equation (3): （3） wherein, denotes the execution of the operations is divided into phases, uses devices, each phase has a delay of no more than minimum total delay, is a subgraph consisting of the th to the th operation, is the grid shape of the devices, is the execution delay of the subgraph on the specified grid, is the total number of operations; Where the initialization of DP is F(0, K+1, 0; tmax) = 0, and the total delay is represented as equation (4): （4） wherein is the total shape of the cluster device mesh; After the two-stage search of intra-op and inter-op, the strategy divides the model into n stages of pipeline parallelism, each stage is mapped to a sub-grid of the hardware device grid, and the intra-op layer tensor partitioning parallel strategy is allocated.

6. The method of claim 1, wherein, In step 3, the optimal mixed parallel offload strategy is converted into a parallel configuration executable by the parallel training framework using a strategy conversion algorithm, the training is executed and the training data and indicators are recorded, the efficiency of large model training and the use of video memory during training are evaluated, and the effect is verified, including: The tensor splitting and data offloading are converted into parallel process groups and data offloading schemes suitable for the Megatron distributed parallel framework developed by NVIDIA by using a strategy conversion algorithm. The algorithm first generates parallel process groups according to the logical topology structure of the hardware device and the hybrid parallel strategy, supporting tensor parallelism, pipeline parallelism and data parallelism; the size and number of each process group are calculated, and the global rank is decomposed into corresponding parallel group ranks by using a formula, and the tensor splitting strategy is mapped into the parallel process groups of Megatron; after the strategy conversion, the generated parallel process groups and data offloading schemes are compared with existing schemes by performing actual measurement in the Megatron framework, and the evaluation indicators include training time, communication overhead and memory utilization, so as to verify the effectiveness of the strategy search.

7. A large model hybrid parallel offloading strategy optimization system in a graphics memory resource limited scene, characterized in that, It comprises: A large model architecture acquisition unit acquires a large model architecture, obtains an HLO computation graph, and acquires hardware computing device information of GPU memory capacity and bandwidth; A large model division unit constructs communication, calculation and memory occupation costs according to the obtained HLO computation graph and hardware computing device information, divides the large model into intra-op and inter-op two layers in the stage of pipeline parallelism, and searches for the optimal hybrid parallel offloading strategy respectively; An optimization verification unit uses a strategy conversion algorithm to convert the optimal hybrid parallel offloading strategy into a parallel configuration executable by a parallel training framework, executes training and records training data and indicators, evaluates the training efficiency of the large model and the memory usage in the training process, and verifies the effect.

8. The system of claim 7, wherein, In the large model architecture acquisition unit, the large model architecture is acquired, the HLO computation graph is obtained, and the hardware computing device information of GPU memory capacity and bandwidth is acquired, including: Firstly, the basic architecture of the large model is acquired through a python script, and the calculation is decomposed into repeatedly stacked operator blocks, and HLO computation graph simulation is performed on all operators in the operator blocks; for each operator in the computation graph, an HLO computation graph is simulated using a class, and the simulation content is divided into three parts: Firstly, the shape of the operator input and output tensor is simulated, which ensures the correctness of the computation graph and the correctness of the strategy simulation; secondly, the splitting strategy of the operator is generated according to the hardware information of the computing device and the corresponding execution overhead is simulated, including communication overhead, calculation overhead, memory overhead and tensor re-sharding overhead, which is used for searching the optimal parallel strategy in the subsequent step; finally, the optimal hybrid parallel strategy is searched according to the computation graph and execution overhead obtained in the previous two steps.

9. The system of claim 7, wherein, In the large model division unit, the communication, calculation and memory occupation costs are constructed according to the obtained HLO computation graph and hardware computing device information, the large model is divided into intra-op and inter-op two layers in the stage of pipeline parallelism, and the optimal hybrid parallel offloading strategy is searched, including: Step 2.

1. According to the obtained HLO calculation graph, hardware computing device information and global communication configuration, a strategy search is performed on the pipeline stages that the model can be divided into within the stage, and an integer linear programming (ILP) algorithm is used to implement the strategy search; Step 2.

2. According to the obtained strategy within the stage, in combination with the view of the hardware device, the division on each stage in the device subgrid is explored, and a dynamic programming (DP) algorithm is used to find the optimal pipeline parallel strategy, that is, the minimum training cost, in combination with the mixed parallel unloading strategy within each stage, to output the optimal mixed parallel unloading strategy.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program, when executed by the processor, implements the steps of the large model mixed parallel unloading strategy optimization method in the display memory resource limited scene in any one of claims 1-6.

Images