Model parallel strategy search method and device, computer equipment and storage medium

Abstract

The invention relates to a model parallel strategy searching method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: initializing a cost estimator, a parameter optimizer and a real-time monitor; acquiring an initial parallel strategy by adopting a parameter optimizer; estimating the execution cost of the initial parallel strategy by adopting a cost estimator; according to the execution cost, a parameter optimizer is adopted to automatically search an optimal parallel strategy parameter based on a preset algorithm and modify the parameter of the initial parallel strategy, and a target parallel strategy is generated; issuing the target parallel strategy to a target model for training; a real-time monitor is adopted to monitor the target model, and whether the target parallel strategy is a parallel strategy adapting to the target model and the hardware state or not is determined; and when the target parallel strategy is the parallel strategy adapting to the target model and the hardware state, taking the target parallel strategy as the optimal parallel strategy. By adopting the method, the model parallel strategy which achieves the best balance between the cost benefit and the memory use can be automatically searched.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning model training, in particular to a model parallel strategy searching method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of deep learning and artificial intelligence, the scale and complexity of models are growing, leading to a sharp increase in computing resource demand. To meet this demand, model parallelization technology has emerged, which can train models in a distributed manner on multiple processors.

[0003] The conventional processing method of model parallelization often relies on domain experts to manually design and adjust parallel strategies. However, given that existing model parallelization involves numerous parameters, manually designing parallel strategies not only consumes time and effort, but also is difficult to adapt to rapidly evolving model architectures and hardware environments. In addition, as the diversity of models and hardware continues to increase, the search space for finding the optimal parallel strategy becomes extremely large; at the same time, modern computing clusters often contain multiple types of processors and accelerators, and traditional manual search methods are difficult to effectively cope with heterogeneous environments.

[0004] Therefore, how to automatically search for a model parallel strategy that achieves the best balance between cost-effectiveness and memory usage is a problem that needs to be solved. SUMMARY

[0005] Therefore, it is necessary to provide a model parallel strategy searching method, device, computer equipment, computer readable storage medium and computer program product that can automatically search for a model parallel strategy that achieves the best balance between cost-effectiveness and memory usage.

[0006] In a first aspect, the present application provides a model parallel strategy searching method, comprising:

[0007] initializing a cost estimator, a parameter optimizer and a real-time monitor;

[0008] obtaining initial data; based on the initial data, using the parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, using the cost estimator to estimate the execution cost of the initial parallel strategy;

[0009] based on the execution cost, using the parameter optimizer to automatically search for optimal parallel strategy parameters in the search space of parallel strategy parameters based on a preset algorithm; based on the optimal parallel strategy parameters, using the parameter optimizer to change the parameters of the initial parallel strategy to generate a target parallel strategy;

[0010] issuing the target parallel strategy to a target model, and the target model trains using the target parallel strategy; using the real-time monitor to monitor the target model and obtain real-time performance data;

[0011] Based on the real-time performance data, the real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy adaptive to the target model and the hardware state; when the target parallel strategy is a parallel strategy adaptive to the target model and the hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0012] In one of the embodiments, based on the initial data, the parameter optimizer is used to obtain an initial parallel strategy, including:

[0013] Based on the initial data, the parameter optimizer is used to confirm a search space of parallel strategy parameters; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters and data parallel parameters;

[0014] The pipeline parallel parameters and the tensor parallel parameters are set to preset values, the preset values being in the search space of the parallel strategy parameters, to obtain initial pipeline parallel parameters and initial tensor parallel parameters;

[0015] The product of the initial pipeline parallel parameters and the initial tensor parallel parameters is calculated, and the ratio between the number of graphic processors in the initial data and the corresponding product value is calculated to obtain initial data parallel parameters;

[0016] Based on the initial pipeline parallel parameters, the initial tensor parallel parameters and the initial data parallel parameters, the initial parallel strategy is obtained.

[0017] In one of the embodiments, according to the initial parallel strategy, the cost estimator is used to estimate the execution cost of the initial parallel strategy, including:

[0018] According to the initial parallel strategy, the cost estimator is used to estimate the calculation time, the communication time and the memory consumption of the initial parallel strategy;

[0019] Based on the calculation time, the communication time and the memory consumption, the total cost of the initial parallel strategy is calculated, and the total cost is taken as the execution cost of the initial parallel strategy.

[0020] In one of the embodiments, the method further includes:

[0021] When the target parallel strategy is not a parallel strategy adaptive to the target model and the hardware state, the target parallel strategy is replaced by the initial parallel strategy, and the steps of estimating the execution cost of the initial parallel strategy according to the initial parallel strategy and the subsequent steps are continued to be executed until a preset maximum iteration number is reached or it is confirmed that the target parallel strategy is a parallel strategy adaptive to the target model and the hardware state, and the corresponding target parallel strategy is taken as the optimal parallel strategy.

[0022] In one of the embodiments, the real-time monitor is used to monitor the target model to obtain the real-time performance data, including:

[0023] The real-time monitor is used to monitor the model training progress of the target model and the state of the hardware running the target model in real time, and collect performance indicators;

[0024] Based on the performance indicators, real-time performance data is obtained.

[0025] In one embodiment, according to the execution cost, the parameter optimizer is used to automatically search for optimal parallel strategy parameters in the search space of the parallel strategy parameters based on a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate a target parallel strategy, including:

[0026] According to the execution cost, the parameter optimizer is used to confirm the optimization target of the initial parallel strategy;

[0027] Based on the optimization target, the parameter optimizer is used to obtain optimal parallel strategy parameters in the search space of the parallel strategy parameters based on a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate an intermediate parallel strategy;

[0028] The parameter optimizer is used to confirm whether the intermediate parallel strategy has reached the optimization target;

[0029] If the intermediate parallel strategy has not reached the optimization target, the intermediate parallel strategy is replaced by the initial parallel strategy, and the subsequent steps of estimating the execution cost of the initial parallel strategy using the cost estimator according to the initial parallel strategy are continued until a preset maximum number of iterations is reached or it is confirmed that the intermediate parallel strategy has reached the optimization target, and the corresponding intermediate parallel strategy is taken as the target parallel strategy;

[0030] If the intermediate parallel strategy has reached the optimization target, the intermediate parallel strategy is taken as the target parallel strategy.

[0031] In a second aspect, the present application also provides a model parallel strategy searching device, including:

[0032] An initial module is used to initialize the cost estimator, the parameter optimizer, and the real-time monitor;

[0033] An estimation module is used to obtain initial data; based on the initial data, the parameter optimizer is used to obtain an initial parallel strategy; and according to the initial parallel strategy, the cost estimator is used to estimate the execution cost of the initial parallel strategy;

[0034] A search module is used to, according to the execution cost, automatically search for optimal parallel strategy parameters in the search space of the parallel strategy parameters based on a preset algorithm using the parameter optimizer; and based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate a target parallel strategy;

[0035] The monitoring module is used to distribute the target parallel strategy to the target model, and the target model is trained using the target parallel strategy; a real-time monitor is used to monitor the target model and obtain real-time performance data.

[0036] The verification module is used to verify, based on real-time performance data and a real-time monitor, whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state; if the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, then the target parallel strategy is taken as the optimal parallel strategy.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0038] Initialize the cost estimator, parameter optimizer, and real-time monitor;

[0039] Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy;

[0040] Based on the execution cost, a parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy;

[0041] The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy. A real-time monitor is used to monitor the target model and obtain real-time performance data.

[0042] Based on real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state. When the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0044] Initialize the cost estimator, parameter optimizer, and real-time monitor;

[0045] Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy;

[0046] Based on the execution cost, a parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy;

[0047] The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy. A real-time monitor is used to monitor the target model and obtain real-time performance data.

[0048] Based on real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state. When the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0049] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0050] Initialize the cost estimator, parameter optimizer, and real-time monitor;

[0051] Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy;

[0052] Based on the execution cost, a parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy;

[0053] The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy. A real-time monitor is used to monitor the target model and obtain real-time performance data.

[0054] Based on real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state. When the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0055] The aforementioned model parallel strategy search method, apparatus, computer equipment, computer-readable storage medium, and computer program product initialize a cost estimator, a parameter optimizer, and a real-time monitor; acquire initial data; based on the initial data, use the parameter optimizer to acquire an initial parallel strategy; according to the initial parallel strategy, use the cost estimator to estimate the execution cost of the initial parallel strategy; based on the execution cost, use the parameter optimizer to automatically search for optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, use the parameter optimizer to change the parameters of the initial parallel strategy to generate a target parallel strategy; distribute the target parallel strategy to the target model, and the target model is trained using the target parallel strategy; use the real-time monitor to monitor the target model and acquire real-time performance data; based on the real-time performance data, use the real-time monitor to confirm whether the target parallel strategy is a parallel strategy adapted to the target model and hardware state; when the target parallel strategy is a parallel strategy adapted to the target model and hardware state, then the target parallel strategy is taken as the optimal parallel strategy. Through the coordinated operation of the cost estimator, parameter optimizer, and real-time monitor, it can not only automatically search for the model parallelism strategy that achieves the best balance between cost-effectiveness and memory usage in the huge search space of optimal parallelism strategies, but also adapt to rapidly evolving model architectures and hardware environments, effectively addressing the challenges of heterogeneous environments. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a diagram illustrating the application environment of a model-parallel strategy search method in one embodiment.

[0058] Figure 2 This is a flowchart illustrating a model-parallel strategy search method in one embodiment;

[0059] Figure 3 This is a flowchart illustrating the model parallel strategy search method in another embodiment;

[0060] Figure 4 This is a schematic diagram illustrating the workflow of each component in a model-parallel strategy search method in one embodiment;

[0061] Figure 5 This is a block diagram of the model parallel policy search device in one embodiment;

[0062] Figure 6This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0065] The model parallel strategy search method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Specifically, terminal 102 or server 104 completes a model parallel strategy search method, which includes: initializing a cost estimator, a parameter optimizer, and a real-time monitor; acquiring initial data; based on the initial data, using the parameter optimizer to obtain an initial parallel strategy; based on the initial parallel strategy, using the cost estimator to estimate the execution cost of the initial parallel strategy; based on the execution cost, using the parameter optimizer to automatically search for optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, using the parameter optimizer to change the parameters of the initial parallel strategy to generate a target parallel strategy; distributing the target parallel strategy to the target model, and the target model is trained using the target parallel strategy; using the real-time monitor to monitor the target model and acquire real-time performance data; based on the real-time performance data, using the real-time monitor to confirm whether the target parallel strategy is a parallel strategy adapted to the target model and hardware state; when the target parallel strategy is a parallel strategy adapted to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0066] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0067] In one exemplary embodiment, such as Figure 2 As shown, a model parallel policy search method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 210. Wherein:

[0068] Step 202: Initialize the cost estimator, parameter optimizer, and real-time monitor.

[0069] Optionally, the device type can be a graphics processing unit (GPU) or a neural network processing unit (NPU). The following description uses a GPU as an example; the NPU is similar.

[0070] The components include: a cost estimator, which predicts the time, money, and computing resources required for model training or inference; a parameter optimizer, which uses the data provided by the cost estimator to automatically search for the optimal parallel strategy parameter configuration using a preset algorithm; and a real-time monitor, which dynamically adjusts the parallel strategy based on real-time feedback to adapt to changes during model training.

[0071] Step 204: Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy.

[0072] The initial data includes model file data, server resource data, and training data; the initial parallel strategy is the preliminary distributed training scheme formulated by the parameter optimizer for the target model after analyzing the initial data. Parallel strategies include data parallelism, pipeline parallelism, and tensor parallelism.

[0073] Optionally, model file data, server resource data, and training data are acquired; a parameter optimizer is used to analyze the model file data, server resource data, and training data; an initial pipeline parallel strategy is obtained based on the model file data; a data parallel strategy and a tensor parallel strategy are obtained based on the server resource data; and the data parallel strategy, pipeline parallel strategy, and tensor parallel strategy are merged to obtain the initial parallel strategy.

[0074] The execution cost includes computation time, communication time, and memory consumption under a parallel strategy. Memory consumption includes model parameters, intermediate activation values, and gradients. Computation time needs to consider whether activation re-enhancing techniques are used during training.

[0075] For example, based on real-time performance data during the training process of the target model using the initial parallel strategy, a cost estimator is used to estimate the computation time, communication time, and memory consumption under the initial parallel strategy; the cost estimator is then used to estimate the total cost of the computation time, communication time, and memory consumption under the initial parallel strategy, where the total cost is the execution cost of the initial parallel strategy.

[0076] Step 206: Based on the execution cost, the parameter optimizer automatically searches for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate the target parallel strategy.

[0077] Optionally, the preset algorithm can be one of the following: genetic algorithm, dynamic programming, or simulated annealing. The optimization objective of the parallel strategy parameters is defined as achieving optimal execution cost and real-time performance.

[0078] The search space for the parallel policy parameters is the set of all possible solutions for the parallel policy parameters. The target parallel policy is the parallel policy that achieves optimal execution cost.

[0079] For example, a parameter optimizer is used with the goal of minimizing execution cost. A genetic algorithm is used to automatically optimize the parallel strategy parameters in the search space of the parallel strategy parameters, ultimately outputting the optimal parallel strategy parameters. Based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy. Specifically, automatic optimization is an iterative process.

[0080] Step 208: The target parallel strategy is distributed to the target model, and the target model is trained using the target parallel strategy; a real-time monitor is used to monitor the target model and obtain real-time performance data.

[0081] Among them, real-time performance data is an indicator collected by the real-time monitor during the model training process to quantitatively evaluate the effectiveness of the current parallel strategy and the health of the system; the target model is a deep learning model that needs to be trained in a distributed manner.

[0082] For example, a real-time monitor is used to continuously collect, analyze, and record various metrics related to the training process of the target model and the status of the hardware running the target model during the training of the target model using a parallel strategy.

[0083] Step 210: Based on real-time performance data, use a real-time monitor to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state; if the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, then the target parallel strategy is taken as the optimal parallel strategy.

[0084] Among them, the optimal parallel strategy is the parallel strategy that adapts to the real-time target model training state and hardware state and achieves the optimal execution cost.

[0085] For example, by referring to various metrics related to the target model training process and the state of the hardware running the target model, it is determined whether the target parallel strategy is a parallel strategy adapted to the target model and the hardware state. When the target parallel strategy is a parallel strategy adapted to the target model and the hardware state, then the target parallel strategy is the parallel strategy that adapts to the real-time target model training state and the hardware state and achieves optimal execution cost, and the target parallel strategy is used as the optimal parallel strategy for subsequent target model training.

[0086] In the above-described model parallel strategy search method, the following steps are taken: initializing the cost estimator, parameter optimizer, and real-time monitor; initial data is acquired; based on the initial data, the parameter optimizer is used to obtain an initial parallel strategy; according to the initial parallel strategy, the cost estimator is used to estimate the execution cost of the initial parallel strategy; based on the execution cost, the parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate a target parallel strategy; the target parallel strategy is distributed to the target model, and the target model is trained using the target parallel strategy; the real-time monitor is used to monitor the target model and acquire real-time performance data; based on the real-time performance data, the real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy adapted to the target model and hardware state; when the target parallel strategy is a parallel strategy adapted to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy. Through the coordinated operation of the cost estimator, parameter optimizer, and real-time monitor, it can not only automatically search for the model parallelism strategy that achieves the best balance between cost-effectiveness and memory usage in the huge search space of optimal parallelism strategies, but also adapt to rapidly evolving model architectures and hardware environments, effectively addressing the challenges of heterogeneous environments.

[0087] In one embodiment, obtaining an initial parallel strategy using a parameter optimizer includes: obtaining an initial parallel strategy based on initial data using the parameter optimizer, including: determining the search space of parallel strategy parameters using the parameter optimizer based on the initial data; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters, and data parallel parameters; setting the pipeline parallel parameters and tensor parallel parameters to preset values, the preset values ​​being in the search space of the parallel strategy parameters, to obtain the initial pipeline parallel parameters and initial tensor parallel parameters; calculating the product of the initial pipeline parallel parameters and initial tensor parallel parameters, calculating the ratio between the number of graphics processors in the initial data and the corresponding product value, to obtain the initial data parallel parameters; and obtaining the initial parallel strategy based on the initial pipeline parallel parameters, initial tensor parallel parameters, and initial data parallel parameters.

[0088] The model file data is a collection of all relevant information describing the structure, parameters, and computational characteristics of the deep learning model to be trained, including data such as the number of model layers and the number of model parameters; the server resource data is detailed information on the hardware configuration, topology, and real-time status of the computing cluster or server that performs the model training task, including data such as the number of graphics processing units (GPUs) and the number of single-click cards; the training data consists of the samples used to train the model and their data-related characteristics and distribution, including the total number of training tokens.

[0089] For example, the parallel strategy parameters include pipeline parallelism parameters, initial tensor parallelism parameters, and data parallelism parameters. Based on the number of model layers, the parameter optimizer determines the search space for the pipeline parallelism parameters as follows:

[0090]

[0091] in, For pipeline parallel parameters, For the number of model layers, additional Must be able to Divisible, denoted as .

[0092] Based on the number of cards per machine, a parameter optimizer is used to confirm that the search space of tensor parallel parameters is reduced proportionally to the number of cards per machine. For example, if there are 8 cards per machine, the range of TP values ​​is [8, 4, 2, 1].

[0093] Based on the number of graphics processing units (GPUs), the search space for data parallelism parameters is determined using a parameter optimizer:

[0094]

[0095] in, For data parallelism parameters, For pipeline parallel parameters, For tensor parallel parameters.

[0096] The parameter optimizer is used to set the pipeline parallelism parameter to 1 and the tensor parallelism parameter to the number of cards per machine; that is, the initial pipeline parallelism parameter is 1 and the initial tensor parallelism parameter is the number of cards per machine.

[0097] In this embodiment, by fully considering factors such as the structural characteristics of the model, server resource status, and training data scale, the search space of the parallel strategy parameters is defined, and the initial pipeline parallel parameters, initial tensor parallel parameters, and initial data parallel parameters are reasonably set based on the initial data. This enables the accurate acquisition of an initial parallel strategy that can better adapt to the training requirements of the target model, laying a solid foundation for subsequent search for the optimal parallel strategy based on real-time performance data and execution cost.

[0098] In one embodiment, the execution cost of the initial parallel strategy is estimated using a cost estimator based on the initial parallel strategy, including: estimating the computation time, communication time, and memory consumption of the initial parallel strategy using the cost estimator based on the initial parallel strategy; calculating the total cost of the initial parallel strategy based on the computation time, communication time, and memory consumption, and using the total cost as the execution cost of the initial parallel strategy.

[0099] In this context, computation time refers to the total time spent by a single device (such as a GPU) performing all its allocated computational operations under a given model parallelism strategy. This primarily includes forward propagation and backpropagation computations. The number of model parameters and the total number of training tokens determine the computational load required to train the target model; given the hardware device type, the required training time can be estimated. Communication time refers to the total time spent by devices transferring data (such as activation values, gradients, and model parameters) for collaborative work under a given model parallelism strategy. Memory consumption refers to the maximum amount of GPU memory required by a single device under a given model parallelism strategy.

[0100] For example, when the device type is GPU, two floating-point operations are required for each parameter of the target model, i.e., calculation When performing a forward propagation, the matrices are first multiplied bitwise, and then added bitwise, representing one forward propagation. Therefore, the computational complexity of the forward propagation is estimated as follows:

[0101]

[0102] If activation recalculation is used to reduce the need for an extra forward pass when activating intermediate memory, and since backpropagation computation is twice that of forward propagation, the estimated computation time in a scenario without activation recalculation is as follows:

[0103]

[0104] In scenarios involving activation recalculation, the estimated computation time is as follows:

[0105]

[0106] Based on the initial data, the cost estimator uses the above method to estimate the computation time of the initial parallel strategy.

[0107] For example, in an all-reduce set communication scenario, if there are n workers, the cost estimator uses the following formula to estimate the communication time of the initial parallel strategy based on the initial data:

[0108]

[0109] For example, based on the initial data, the cost estimator uses the following formula to estimate the communication time of the initial parallel strategy:

[0110]

[0111] Where R is the number of stacked transformer layers in the target model, N is the number of attention heads, D is the dimension of the attention heads, B (batch size) is the batch size, and S (sequence length) is the length of the input sequence.

[0112] Specifically, the number of stacked transformer layers in the target model is the model depth, i.e., the number of times the Transformer block is repeated, which is the most critical factor in determining whether to use pipelined parallelism; the number of attention heads is the number of "heads" in a multi-head attention mechanism, each head is responsible for understanding the input sequence from a different perspective, and is an important indicator of tensor parallelism; the dimension of the attention heads is the vector dimension of each attention head, which is another important indicator of tensor parallelism; the batch size is the total number of training samples processed in one iteration, which is related to data parallelism; and the length of the input sequence is the number of tokens contained in the input text, which may introduce the requirement of sequence parallelism.

[0113] For example, the total cost of the initial parallel strategy is calculated based on computation time, communication time, and memory consumption, using the following formula:

[0114]

[0115] in, This is the balance coefficient for multiple objectives.

[0116] In this embodiment, by analyzing the three key factors of computation time, communication time, and memory consumption in detail based on actual data, the cost estimator can more accurately reflect the cost of the initial parallel strategy in actual execution, taking into account the structural characteristics of the model, the status of hardware resources, and the scale of training data. This provides a reliable basis for subsequent optimal parallel strategy search and can adapt to the cost estimation needs of different model architectures and hardware environments, providing strong support for the optimization of model parallel strategies.

[0117] In one embodiment, the method further includes: when the target parallel strategy is not a parallel strategy adapted to the target model and hardware state, replacing the target parallel strategy with the initial parallel strategy, continuing to execute the steps according to the initial parallel strategy, using a cost estimator to estimate the execution cost of the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or it is confirmed that the target parallel strategy is a parallel strategy adapted to the target model and hardware state, and taking the corresponding target parallel strategy as the optimal parallel strategy.

[0118] The preset number of iterations is determined based on a preset algorithm. The optimal parallel strategy should be the one that achieves the best execution cost and adaptability to the target model and hardware state.

[0119] For example, the optimal parallel strategy is used for the final training of the target model.

[0120] In this embodiment, when the real-time monitor detects a deviation between the target parallel strategy and the actual training state, the system triggers a parameter adjustment process, automatically adjusts the parameters, and seamlessly integrates the updated strategy into subsequent training stages. Based on the real-time performance data of the target model during training, iterative optimization of parallel strategy parameters that do not meet the current model and hardware state can be performed. This effectively solves the problem of decreased parallel efficiency caused by fluctuations in hardware resources or changes in model architecture during model training, and can continuously approach the optimal solution within a preset maximum number of iterations.

[0121] In one embodiment, a real-time monitor is used to monitor the target model and obtain real-time performance data, including: using a real-time monitor to monitor the model training progress of the target model and the status of the hardware running the target model in real time, and collecting performance indicators; and obtaining real-time performance data based on the performance indicators.

[0122] Among these, performance metrics are the core feedback signals driving the entire iterative optimization process. Performance metrics include throughput, latency, and memory usage, among others.

[0123] In this embodiment, a real-time monitor is used to capture changes in the training progress of the target model and the hardware status in real time, and collects multiple performance indicators to accurately obtain real-time performance data. At the same time, the real-time performance data is used as a key basis for subsequent strategy adjustments, which can ensure that the parallel strategy can dynamically adapt to various changes in the model training process, thereby effectively improving the efficiency and stability of model training.

[0124] In one embodiment, based on execution cost, a parameter optimizer automatically searches for optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate a target parallel strategy, including: based on execution cost, the parameter optimizer confirms the optimization objective of the initial parallel strategy; based on the optimization objective, the parameter optimizer obtains the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate an intermediate parallel strategy; the parameter optimizer confirms whether the intermediate parallel strategy has reached the optimization objective; if the intermediate parallel strategy has not reached the optimization objective, the intermediate parallel strategy is replaced with the initial parallel strategy, and the execution cost of the initial parallel strategy is estimated using a cost estimator, and subsequent steps continue to be executed until a preset maximum number of iterations is reached or it is confirmed that the intermediate parallel strategy has reached the optimization objective, and the corresponding intermediate parallel strategy is used as the target parallel strategy; if the intermediate parallel strategy has reached the optimization objective, the intermediate parallel strategy is used as the target parallel strategy.

[0125] Among them, the optimization objective is the final direction and evaluation criterion of the entire iterative optimization process. Based on the execution cost, the parameter optimizer is used to confirm that the optimization objective of the initial parallel strategy is the optimal one in terms of execution cost.

[0126] In this embodiment, by explicitly defining the optimization objective as optimal execution cost, the parameter optimizer conducts an efficient search within a predetermined parallel strategy parameter search space to accurately locate the optimal parallel strategy parameters and generate an intermediate parallel strategy. Simultaneously, to ensure the effectiveness and optimality of the strategy, it further verifies whether the intermediate parallel strategy achieves the preset optimization objective. If not, the intermediate parallel strategy is reverted to its initial state, and the search and adjustment process is restarted, iterating continuously until the preset maximum number of iterations is met or it is confirmed that the intermediate parallel strategy has successfully achieved the optimization objective. At this point, the intermediate parallel strategy is established as the target parallel strategy. This strategy ensures the continuous optimization and accurate adaptation of the parallel strategy, providing efficient and stable support for model training.

[0127] Next reference Figure 3 The present application’s model parallel strategy search method will be illustrated with a specific embodiment.

[0128] Step 1: Obtain the parameter optimizer, cost estimator, and real-time monitor;

[0129] Step 2: Obtain model files, server resources, and training data;

[0130] Step 3: Input the model file, server resources, and training data into the parameter optimizer to generate the initial parallel strategy.

[0131] Step 4: The parameter optimizer iteratively optimizes the initial parallel policy based on the initial policy using an algorithm.

[0132] Step 5: During the iterative optimization process, a cost estimator is used to estimate the execution cost of each parallel strategy. The optimization objective is to optimize the execution cost, and the optimal parallel strategy is generated.

[0133] Step 6: Deploy the optimal parallel strategy to the target model for execution, and use a real-time monitor to monitor the training progress of the target model and the status of the hardware running the target model in real time.

[0134] Step 7: If the optimal parallel strategy is not a parallel strategy that is suitable for the target model and hardware state, then return to step 4.

[0135] like Figure 4As shown, the parameter optimizer generates an initial parallel strategy, which is then iterated through the algorithm to generate the optimal parallel strategy. The cost estimator estimates the execution cost of each parallel strategy during the algorithm iteration process and performs algorithm iteration based on the execution cost. After the optimal parallel strategy is issued and executed, the real-time monitor monitors the target model and its hardware status in real time, and modifies the parameters during the algorithm iteration process based on the target model and its hardware status.

[0136] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0137] Based on the same inventive concept, this application also provides a model parallel policy search apparatus for implementing the model parallel policy search method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the model parallel policy search apparatus provided below can be found in the limitations of the model parallel policy search method described above, and will not be repeated here.

[0138] In one exemplary embodiment, such as Figure 5 As shown, a model-parallel policy search device 500 is provided, including: an initialization module 502, an estimation module 504, a search module 506, a monitoring module 508, and a confirmation module 510, wherein:

[0139] Initial module 502 is used to initialize the cost estimator, parameter optimizer, and real-time monitor.

[0140] The estimation module 504 is used to obtain initial data; based on the initial data, the parameter optimizer is used to obtain the initial parallel strategy; according to the initial parallel strategy, the cost estimator is used to estimate the execution cost of the initial parallel strategy.

[0141] The search module 506 is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters based on the execution cost and a preset algorithm using a parameter optimizer; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate the target parallel strategy.

[0142] The monitoring module 508 is used to distribute the target parallel strategy to the target model, and the target model is trained using the target parallel strategy; a real-time monitor is used to monitor the target model and obtain real-time performance data.

[0143] The confirmation module 510 is used to confirm, based on real-time performance data and using a real-time monitor, whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state; when the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0144] In one embodiment, the estimation module 504 is further configured to, based on initial data, use a parameter optimizer to determine the search space of parallel strategy parameters; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters, and data parallel parameters; set the pipeline parallel parameters and tensor parallel parameters to preset values, and obtain the initial pipeline parallel parameters and initial tensor parallel parameters from the search space of the parallel strategy parameters; calculate the product of the initial pipeline parallel parameters and initial tensor parallel parameters, calculate the ratio between the number of graphics processors in the initial data and the corresponding product value, and obtain the initial data parallel parameters; and obtain the initial parallel strategy based on the initial pipeline parallel parameters, initial tensor parallel parameters, and initial data parallel parameters.

[0145] In one embodiment, the estimation module 504 is further configured to estimate the computation time, communication time, and memory consumption of the initial parallel strategy using a cost estimator based on real-time performance data; calculate the total cost of the initial parallel strategy based on the computation time, communication time, and memory consumption, and use the total cost as the execution cost of the initial parallel strategy.

[0146] In one embodiment, the confirmation module 510 is further configured to replace the target parallel strategy with the initial parallel strategy when the target parallel strategy is not a parallel strategy adapted to the target model and hardware state, and continue to execute the steps of estimating the execution cost of the initial parallel strategy using a cost estimator according to the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or the target parallel strategy is confirmed to be a parallel strategy adapted to the target model and hardware state, and the corresponding target parallel strategy is taken as the optimal parallel strategy.

[0147] In one embodiment, the monitoring module 508 is further configured to use a real-time monitor to monitor the model training progress of the target model and the status of the hardware running the target model in real time, and collect performance indicators; based on the performance indicators, obtain real-time performance data.

[0148] In one embodiment, the search module 506 is further configured to: determine the optimization objective of the initial parallel strategy using a parameter optimizer based on the execution cost; based on the optimization objective, the parameter optimizer obtains the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate an intermediate parallel strategy; the parameter optimizer confirms whether the intermediate parallel strategy has achieved the optimization objective; if the intermediate parallel strategy has not achieved the optimization objective, the intermediate parallel strategy is replaced with the initial parallel strategy, and the execution cost of the initial parallel strategy is estimated using a cost estimator, and subsequent steps are continued until the preset maximum number of iterations is reached or the intermediate parallel strategy is confirmed to have achieved the optimization objective, and the corresponding intermediate parallel strategy is used as the target parallel strategy; if the intermediate parallel strategy has achieved the optimization objective, the intermediate parallel strategy is used as the target parallel strategy.

[0149] Each module in the aforementioned parallel strategy search device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0150] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores real-time performance data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a model-parallel strategy search method.

[0151] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0153] Initialize the cost estimator, parameter optimizer, and real-time monitor;

[0154] Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy;

[0155] Based on the execution cost, a parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy;

[0156] The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy. A real-time monitor is used to monitor the target model and obtain real-time performance data.

[0157] Based on real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state. When the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0158] In one embodiment, when the processor executes the computer program, it further performs the following steps: based on initial data, a parameter optimizer is used to determine the search space of parallel strategy parameters; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters, and data parallel parameters; the pipeline parallel parameters and tensor parallel parameters are set to preset values, and the preset values ​​are in the search space of the parallel strategy parameters to obtain the initial pipeline parallel parameters and initial tensor parallel parameters; the product of the initial pipeline parallel parameters and initial tensor parallel parameters is calculated, and the ratio between the number of graphics processors in the initial data and the corresponding product value is calculated to obtain the initial data parallel parameters; based on the initial pipeline parallel parameters, initial tensor parallel parameters, and initial data parallel parameters, an initial parallel strategy is obtained.

[0159] In one embodiment, when the processor executes the computer program, it further performs the following steps: based on the initial parallel strategy, it uses a cost estimator to estimate the computation time, communication time, and memory consumption of the initial parallel strategy; based on the computation time, communication time, and memory consumption, it calculates the total cost of the initial parallel strategy and uses the total cost as the execution cost of the initial parallel strategy.

[0160] In one embodiment, when the processor executes the computer program, it further implements the following steps: when the target parallel strategy is not a parallel strategy adapted to the target model and hardware state, the target parallel strategy is replaced with the initial parallel strategy, and the execution of the initial parallel strategy is continued according to the initial parallel strategy, using a cost estimator to estimate the execution cost of the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or it is confirmed that the target parallel strategy is a parallel strategy adapted to the target model and hardware state, and the corresponding target parallel strategy is taken as the optimal parallel strategy.

[0161] In one embodiment, when the processor executes the computer program, it also performs the following steps: using a real-time monitor to monitor the model training progress of the target model and the status of the hardware running the target model in real time, and collect performance metrics; and based on the performance metrics, obtain real-time performance data.

[0162] In one embodiment, when the processor executes the computer program, it further performs the following steps: Based on the execution cost, a parameter optimizer is used to determine the optimization objective of the initial parallel strategy; based on the optimization objective, the parameter optimizer obtains the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate an intermediate parallel strategy; the parameter optimizer is used to determine whether the intermediate parallel strategy has achieved the optimization objective; if the intermediate parallel strategy has not achieved the optimization objective, the intermediate parallel strategy is replaced with the initial parallel strategy, and the execution of the steps following the initial parallel strategy—estimated by a cost estimator based on the initial parallel strategy and subsequent steps—continues until a preset maximum number of iterations is reached or the intermediate parallel strategy is confirmed to have achieved the optimization objective, and the corresponding intermediate parallel strategy is used as the target parallel strategy; if the intermediate parallel strategy has achieved the optimization objective, the intermediate parallel strategy is used as the target parallel strategy.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0164] Initialize the cost estimator, parameter optimizer, and real-time monitor;

[0165] Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy;

[0166] Based on the execution cost, a parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy;

[0167] The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy. A real-time monitor is used to monitor the target model and obtain real-time performance data.

[0168] Based on real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state. When the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0169] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: based on initial data, a parameter optimizer is used to determine the search space of parallel strategy parameters; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters, and data parallel parameters; the pipeline parallel parameters and tensor parallel parameters are set to preset values, and the preset values ​​are in the search space of the parallel strategy parameters to obtain the initial pipeline parallel parameters and initial tensor parallel parameters; the product of the initial pipeline parallel parameters and initial tensor parallel parameters is calculated, and the ratio between the number of graphics processors in the initial data and the corresponding product value is calculated to obtain the initial data parallel parameters; based on the initial pipeline parallel parameters, initial tensor parallel parameters, and initial data parallel parameters, an initial parallel strategy is obtained.

[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: estimating the computation time, communication time, and memory consumption of the initial parallel strategy using a cost estimator; calculating the total cost of the initial parallel strategy based on the computation time, communication time, and memory consumption, and using the total cost as the execution cost of the initial parallel strategy.

[0171] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when the target parallel strategy is not a parallel strategy adapted to the target model and hardware state, the target parallel strategy is replaced with the initial parallel strategy, and the execution of the initial parallel strategy is continued according to the initial parallel strategy, using a cost estimator to estimate the execution cost of the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or it is confirmed that the target parallel strategy is a parallel strategy adapted to the target model and hardware state, and the corresponding target parallel strategy is taken as the optimal parallel strategy.

[0172] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: using a real-time monitor to monitor the model training progress of the target model and the status of the hardware running the target model in real time, and collect performance metrics; and based on the performance metrics, obtain real-time performance data.

[0173] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: Based on the execution cost, a parameter optimizer is used to determine the optimization objective of the initial parallel strategy; based on the optimization objective, the parameter optimizer obtains the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate an intermediate parallel strategy; the parameter optimizer is used to determine whether the intermediate parallel strategy has achieved the optimization objective; if the intermediate parallel strategy has not achieved the optimization objective, the intermediate parallel strategy is replaced with the initial parallel strategy, and the execution of the steps following the initial parallel strategy—including estimating the execution cost of the initial parallel strategy using a cost estimator—continues until a preset maximum number of iterations is reached or the intermediate parallel strategy is confirmed to have achieved the optimization objective, and the corresponding intermediate parallel strategy is used as the target parallel strategy; if the intermediate parallel strategy has achieved the optimization objective, the intermediate parallel strategy is used as the target parallel strategy.

[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0175] Initialize the cost estimator, parameter optimizer, and real-time monitor;

[0176] Obtain initial data; based on the initial data, use a parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use a cost estimator to estimate the execution cost of the initial parallel strategy;

[0177] Based on the execution cost, a parameter optimizer is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer is used to change the parameters of the initial parallel strategy to generate the target parallel strategy;

[0178] The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy. A real-time monitor is used to monitor the target model and obtain real-time performance data.

[0179] Based on real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state. When the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

[0180] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: based on initial data, a parameter optimizer is used to determine the search space of parallel strategy parameters; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters, and data parallel parameters; the pipeline parallel parameters and tensor parallel parameters are set to preset values, and the preset values ​​are in the search space of the parallel strategy parameters to obtain the initial pipeline parallel parameters and initial tensor parallel parameters; the product of the initial pipeline parallel parameters and initial tensor parallel parameters is calculated, and the ratio between the number of graphics processors in the initial data and the corresponding product value is calculated to obtain the initial data parallel parameters; based on the initial pipeline parallel parameters, initial tensor parallel parameters, and initial data parallel parameters, an initial parallel strategy is obtained.

[0181] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: estimating the computation time, communication time, and memory consumption of the initial parallel strategy using a cost estimator; calculating the total cost of the initial parallel strategy based on the computation time, communication time, and memory consumption, and using the total cost as the execution cost of the initial parallel strategy.

[0182] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when the target parallel strategy is not a parallel strategy adapted to the target model and hardware state, the target parallel strategy is replaced with the initial parallel strategy, and the execution of the initial parallel strategy is continued according to the initial parallel strategy, using a cost estimator to estimate the execution cost of the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or it is confirmed that the target parallel strategy is a parallel strategy adapted to the target model and hardware state, and the corresponding target parallel strategy is taken as the optimal parallel strategy.

[0183] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: using a real-time monitor to monitor the model training progress of the target model and the status of the hardware running the target model in real time, and collect performance metrics; and based on the performance metrics, obtain real-time performance data.

[0184] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: Based on the execution cost, a parameter optimizer is used to determine the optimization objective of the initial parallel strategy; based on the optimization objective, the parameter optimizer obtains the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate an intermediate parallel strategy; the parameter optimizer is used to determine whether the intermediate parallel strategy has achieved the optimization objective; if the intermediate parallel strategy has not achieved the optimization objective, the intermediate parallel strategy is replaced with the initial parallel strategy, and the execution of the steps following the initial parallel strategy—including estimating the execution cost of the initial parallel strategy using a cost estimator—continues until a preset maximum number of iterations is reached or the intermediate parallel strategy is confirmed to have achieved the optimization objective, and the corresponding intermediate parallel strategy is used as the target parallel strategy; if the intermediate parallel strategy has achieved the optimization objective, the intermediate parallel strategy is used as the target parallel strategy.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model-parallel strategy search method, characterized in that, The method includes: Initialize the cost estimator, parameter optimizer, and real-time monitor; Obtain initial data; based on the initial data, use the parameter optimizer to obtain an initial parallel strategy; according to the initial parallel strategy, use the cost estimator to estimate the execution cost of the initial parallel strategy; Based on the execution cost, the parameter optimizer automatically searches for the optimal parallel strategy parameters in the search space of parallel strategy parameters using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate the target parallel strategy. The target parallel strategy is distributed to the target model, which is then trained using the target parallel strategy; the real-time monitor is used to monitor the target model and obtain real-time performance data. Based on the real-time performance data, a real-time monitor is used to confirm whether the target parallel strategy is a parallel strategy that adapts to the target model and hardware state; when the target parallel strategy is a parallel strategy that adapts to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

2. The method according to claim 1, characterized in that, The step of obtaining an initial parallel strategy based on the initial data using the parameter optimizer includes: Based on the initial data, the parameter optimizer is used to determine the search space for the parallel strategy parameters; the parallel strategy parameters include pipeline parallel parameters, initial tensor parallel parameters, and data parallel parameters. The pipeline parallelism parameters and tensor parallelism parameters are set to preset values, and the initial pipeline parallelism parameters and initial tensor parallelism parameters are obtained in the search space of the parallelism strategy parameters. Calculate the product of the initial pipeline parallel parameters and the initial tensor parallel parameters, and calculate the ratio between the number of graphics processors in the initial data and the corresponding product value to obtain the initial data parallel parameters; Based on the initial pipeline parallel parameters, the initial tensor parallel parameters, and the initial data parallel parameters, the initial parallel strategy is obtained.

3. The method according to claim 1, characterized in that, The step of estimating the execution cost of the initial parallel strategy using the cost estimator, based on the initial parallel strategy, includes: Based on the initial parallel strategy, the cost estimator is used to estimate the computation time, communication time, and memory consumption of the initial parallel strategy. Based on the computation time, communication time, and memory consumption, the total cost of the initial parallel strategy is calculated, and the total cost is used as the execution cost of the initial parallel strategy.

4. The method according to claim 1, characterized in that, The method further includes: When the target parallel strategy is not a parallel strategy adapted to the target model and hardware state, the target parallel strategy is replaced with the initial parallel strategy, and the execution continues. The cost estimator is used to estimate the execution cost of the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or it is confirmed that the target parallel strategy is a parallel strategy adapted to the target model and hardware state. The corresponding target parallel strategy is then taken as the optimal parallel strategy.

5. The method according to claim 1, characterized in that, The step of using the real-time monitor to monitor the target model and obtain real-time performance data includes: The real-time monitor is used to monitor the training progress of the target model and the status of the hardware running the target model in real time, and to collect performance metrics. Based on the aforementioned performance metrics, obtain real-time performance data.

6. The method according to claim 1, characterized in that, The step of automatically searching for optimal parallel strategy parameters in the search space of parallel strategy parameters using the parameter optimizer based on a preset algorithm, according to the execution cost; and using the parameter optimizer to change the parameters of the initial parallel strategy to generate a target parallel strategy based on the optimal parallel strategy parameters, includes: Based on the execution cost, the parameter optimizer is used to determine the optimization objective of the initial parallel strategy; Based on the optimization objective, the parameter optimizer obtains the optimal parallel strategy parameters in the search space of parallel strategy parameters based on a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate an intermediate parallel strategy. The parameter optimizer is used to confirm whether the intermediate parallel strategy has achieved the optimization objective. If the intermediate parallel strategy fails to achieve the optimization objective, the intermediate parallel strategy is replaced with the initial parallel strategy, and the execution of the initial parallel strategy is continued, using the cost estimator to estimate the execution cost of the initial parallel strategy and subsequent steps, until the preset maximum number of iterations is reached or it is confirmed that the intermediate parallel strategy has achieved the optimization objective, and the corresponding intermediate parallel strategy is taken as the target parallel strategy. If the intermediate parallel strategy has achieved the optimization objective, the intermediate parallel strategy shall be adopted as the target parallel strategy.

7. A model-parallel strategy search device, characterized in that, The device includes: The initialization module is used to initialize the cost estimator, parameter optimizer, and real-time monitor. An estimation module is used to obtain initial data; based on the initial data, the parameter optimizer is used to obtain an initial parallel strategy; and according to the initial parallel strategy, the cost estimator is used to estimate the execution cost of the initial parallel strategy. The search module is used to automatically search for the optimal parallel strategy parameters in the search space of parallel strategy parameters based on the execution cost and the parameter optimizer using a preset algorithm; based on the optimal parallel strategy parameters, the parameter optimizer changes the parameters of the initial parallel strategy to generate the target parallel strategy. The monitoring module is used to distribute the target parallel strategy to the target model, and the target model is trained using the target parallel strategy; the real-time monitor is used to monitor the target model and obtain real-time performance data. The confirmation module is used to confirm, based on the real-time performance data, whether the target parallel strategy is a parallel strategy adapted to the target model and hardware state using a real-time monitor; when the target parallel strategy is a parallel strategy adapted to the target model and hardware state, the target parallel strategy is taken as the optimal parallel strategy.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images