Large model training resource evaluation and strategy recommendation method and system

Abstract

The invention discloses a large model training resource evaluation and strategy recommendation method, which comprises the following steps of: 1) constructing a large model database, and storing model state data of different types and sizes; the model database supports expansion and is used for synchronously updating the latest open source large model; 2) constructing a calculation overhead evaluation database of different distributed strategies; defining the video memory efficiency as an index for measuring the degree of storing redundant information on each video card; the calculation efficiency is defined as the proportion of the time for measuring the effective calculation of the graphics card, namely the time for updating the model state and the time for communication; 3) scanning existing computing power resources, traversing the strategy priority queue, and selecting a first hit strategy as a recommendation strategy; aiming at the current available computing resources, recommending a proper distributed training strategy, and giving a super parameter of training recommendation; and according to the selected large model type, estimating the training overhead, scanning node computing resources, giving a training strategy, automatically constructing a configuration file of a training framework, and cooperating with a training platform to realize efficient training of the large model.

Description

Technical Field

[0001] The present invention relates to the field of large model training framework, and in particular to a method and system for recommending a large model distributed training strategy. Background Art

[0002] In recent years, with the continuous growth of computing power and data scale, large models (LLMs) have developed rapidly in the field of artificial intelligence. LLMs usually contain hundreds of millions or even trillions of parameters. They can learn complex semantic relationships and language patterns through unsupervised learning of large amounts of data, and show strong performance in various tasks. LLMs have made breakthrough progress in the field of natural language processing (NLP), and have achieved remarkable results in tasks such as machine translation, text generation, and question-answering systems. For example, the Llama 2 model continues the Transformer's decoder-only architecture and uses Pre-normalization, SwiGLU activation function, rotational embedding encoding (RoPE), longer context, and grouped query attention mechanism. Google AI's LaMDA model has reached a new level in machine translation tasks and can achieve high-quality translation; OpenAI's GPT-3 and ChatGPT models can generate realistic text formats such as poetry, code, scripts, etc.; Facebook AI's Bard model can answer open-ended questions and provide relevant background information. LLMs have also made progress in the field of computer vision (CV), and have achieved good results in tasks such as image classification, image generation, and object detection. For example, Google AI's VisionTransformer model has set a new record in image classification tasks; OpenAI's DALL-E model can generate realistic images; Facebook AI's Detectron2 model has made new progress in object detection tasks. LLM has also been applied in other fields, such as medicine, finance, and law. For example, LLM can be used for medical diagnosis, financial risk prediction, and legal document generation.

[0003] However, the training of LLM requires a large amount of data and computing resources. For example, the training data set size of GPT-3 is 500GB, and the training time is about 100 days; the training data set size of LaMDA is 1.56TB, and the training time is about 200 days. Due to the huge computing power overhead of large models, extremely high requirements are placed on computing resources. This is a huge challenge for small and medium-sized enterprises and individual developers, because they often do not have enough computing power resources to train large models and cannot fully utilize the advantages of large models. Large enterprises often have a large amount of computing power resources. However, these computing power resources are often idle and not fully utilized.

[0004] Therefore, how to efficiently use limited computing power to train large models, or effectively utilize redundant computing power to improve training efficiency has become an urgent and key issue.

[0005] Researchers have proposed a variety of solutions, including: Data parallelism: Divide the data into multiple parts and train them in parallel on multiple devices. This is the simplest distributed training strategy, but the communication overhead is large.

[0006] Model parallelism: Divide the model into multiple parts and train them in parallel on multiple devices. This can effectively reduce communication overhead, but requires additional synchronization operations.

[0007] Pipeline parallelism: The training process is broken down into multiple stages and trained serially on multiple devices. The communication overhead is small, but the efficiency is not as good as data parallelism and model parallelism.

[0008] However, how to dynamically adjust and combine the above-mentioned distributed training strategies according to the existing computing power situation has become a new problem. Therefore, it is necessary to design a method and system that can dynamically recommend large-model distributed training strategies based on the computing power status.

[0009] CN2024106679039 discloses a large model training and a question-answering method, device and equipment based on the large model. The large model training method includes: using a target large model to obtain original thinking chain data based on a question sample; using an advanced large model to obtain improved thinking chain data based on the original thinking chain data; the improved thinking chain data is obtained after the advanced large model improves the logical reasoning process of the original thinking chain data, and the logical reasoning ability of the advanced large model is higher than that of the target large model; based on the question sample and the improved thinking chain data, the target large model is trained. The disclosure can improve the logical reasoning ability of the large model, thereby improving the accuracy of question-answering based on the large model.

[0010] CN2024108176034 discloses a construction and training method, an audio output method and an application of a large speech model. The construction and training method includes: inputting a pre-training set containing pre-training demand data and corresponding pre-training response data into the encoding module of the large speech model, mapping the pre-training demand into a pre-training demand vector to form a pre-training intermediate representation set and sending it to the processing module in the large speech model; the processing module obtains a response result based on the pre-training demand vector and sends it together with the pre-training intermediate representation set into the discriminant module of the large speech model; the pre-training intermediate representation set enters the discriminator in the discriminant module, and the discriminant module sends each response result into the generator of the discriminant module to generate a corresponding audio signal and then enters the discriminator; the discriminator calculates the authenticity score of each audio signal and the loss function of the large speech model and then optimizes it.

[0011] CN2024104385327 discloses a parameter automatic search method, device and electronic device for large model training, including: obtaining a parameter configuration file, which includes a large model training framework name, multiple parameters and a parameter interval of each parameter; determining a target model training framework according to the name of the large model training framework, and determining the training process of all configuration combinations according to the target model training framework, multiple parameters and the parameter interval of each parameter; starting the training process of all configuration combinations, and determining the optimal parameter combination for large model training from the training results of the training process of all configuration combinations based on evaluation indicators. Thus, the optimal parameter configuration combination can be obtained by enumerating and training the parameter configuration combination through the target model training framework, which solves the problem that the current process of determining the optimal parameter configuration is cumbersome and time-consuming, resulting in a long model development cycle, improves the efficiency of users in determining the optimal parameter configuration, and reduces development costs. Summary of the invention

[0012] The technical problem to be solved by the present invention is how to reasonably use the existing computing power when the computing power is limited for large model training, and to quantify the calculation to give a solution and configuration for a reasonable distributed training strategy.

[0013] In order to solve the above technical problems, the technical solution proposed by the present invention is: a method and system for large model training resource evaluation and strategy recommendation, comprising the following steps:

[0014] Step 1: Build a large model database to save model status data of different types and sizes; model types include LLaMA2, Baichuan2, QWen, ChatGLM2 and Mistral. The model status includes: the model's parameter quantity, the model's gradient data quantity, the model's optimizer data quantity and the model's intermediate activation data quantity. The model database supports expansion for synchronously updating the latest open source large models. Specifically:

[0015] Collect the model information of LLaMA2, Baichuan2, QWen, ChatGLM2 and Mistral. For each model, sort out the model parameters of the Embedding layer and deEmbding layer in its structure and record them as P e and P de , and the number of parameters P of the transformer layer t , and the number of layers of transformers, then the overall number of parameters of the model is P = P e +P de +P t*Layers. The parameter quantity P is the total data volume of the model parameters. During the training process, each parameter training will generate a gradient value, so the total gradient value data volume is also P. Similarly, the optimizer will also generate 2P data during the training process. These data volumes together are the storage overhead cost of the model.

[0016] At the same time, the shape of the output tensor of each layer is sorted out, and the amount of intermediate activation data P is accumulated and calculated act ; During the training process, each parameter corresponds to a gradient value, so the data volume of the model gradient is P. The model optimizer uses the Adam algorithm. In this algorithm, each parameter corresponds to two variables, so the data volume of the optimizer is 2P. Since the current industry consensus is to use mixed precision training by default, that is, model parameters, intermediate activations, and gradients use fp16 or BF16, and optimizer variables use fp32. In addition, the optimizer will also store a parameter backup of fp32 for parameter updates during training to avoid errors caused by numerical overflow during mixed precision training as much as possible. fp16 is two bytes and fp32 is 4 bytes. Therefore, the storage cost of the overall model state is:

[0017] cost=2P+2P+2P act +4*2P+4P=16P+2P act

[0018] For the sake of recording, calculate scale act =P act / P, so:

[0019] cost = (16 + scale act )*P

[0020] The above information is constructed in the following dictionary format and stored as a json file:

[0021] {

[0022] Model_type: (llama, baichuan, qwen, chatglm, mistral...),

[0023] Model_size: (6B, 7B, 13B, 14B, 33B, 70B, 140B...),

[0024] Paramter:P,

[0025] Activate:scale act ,

[0026] GPU_Cost: cost,

[0027] mixPrecision:True

[0028] };

[0029] Step 2: Construct a computational cost evaluation database for different distributed strategies. Define memory efficiency as an indicator to measure the degree of redundant information stored on each graphics card. Define computational efficiency as the ratio of the time the graphics card uses for effective computation (i.e., the time the model state is updated) to the time used for communication. Generally speaking, the higher the memory efficiency, the lower the computational efficiency (the reduced redundant information must be compensated by communication), which is suitable for scenarios with tight computing power; conversely, the lower the memory efficiency, the higher the computational efficiency, which is suitable for scenarios with ample computing power. Triggered by computational efficiency and memory efficiency, the memory and communication overhead of the three stages of the data parallel strategy are analyzed, and the strategy priority queue is designed based on this. At the same time, from the perspective of engineering implementation, the peak memory and peak memory overhead of each strategy during training are calculated to construct a cost evaluation database. The details are as follows:

[0030] Step 201: Construct a distributed strategy priority queue based on the data parallel distributed algorithm ZERO. Data parallelism requires running a complete model instance on each graphics card. However, when the model is large and the graphics memory of a single card cannot hold it, the model state needs to be split, with each card responsible for different parts, and communication is used to simulate the training process of a single card running a single instance. ZERO has three strategies: ZERO-1, ZERO-2, and ZERO-3. Assume that there are N graphics cards, and the graphics memory of each graphics card is M. ZERO-1 splits the optimizer in the model state into N parts, and each card stores the optimizer variables corresponding to 1 / N parameters. ZERO-2 splits the optimizer and gradient data into N parts, and ZERO-3 splits the optimizer, gradient, and model parameters into N parts. The evolution of data parallel distributed strategies from ZERO-1 to ZERO-3 is to balance the graphics memory efficiency and computing efficiency. In ZERO-1, only the optimizer data is split. The model instance in each graphics card saves the complete model parameters and gradient data. The proportion of its model state in the video memory is (12P / N+4P) / M. From the horizontal comparison of model instances, the model parameters and gradient data are redundant, that is, the 4P / M part is redundant. Similarly, ZERO-2 saves the complete model parameters, and the proportion of its model state in the video memory is (14P / N+2P) / M, which is also redundant. ZERO-3 has a more complete split, and its model state accounts for 16P / (N*M) in the video memory, without redundancy. From the perspective of computing efficiency, the ZERO-1 strategy only needs to synchronize gradients (all-reduce) and parameters (all-gather) between different instances during training. The total communication volume is 2P(N-1) and the total communication volume is 4P(N-1). The ZERO-2 strategy splits the gradient, so the gradient synchronization needs to be completed when calculating the gradient in reverse. N scatter-reduces need to be performed, and the total communication volume is: N(2P / N*(N-1))=2P(N-1). In addition, the parameters need to be synchronized (all-gather), and the total communication volume is 2P(N-1), and the total communication volume is 4P(N-1). ZERO-3 adds parameter synchronization communication (scatter) in the forward process based on ZERO-2, and the communication volume is 2P(N-1), and the total communication volume increases to 6P(N-1). In terms of computational efficiency, ZERO-3 is the lowest. In addition, although the communication volume of ZERO2 and ZERO-1 is equal, the communication operation of ZERO-2 is interspersed in the process of gradient calculation, which introduces more context switching costs, so its computational efficiency is lower than that of ZERO-1.From the perspective of training speed, the present invention gives priority to the strategy with the highest computing efficiency. In combination with the actual situation of limited idle computing power, the present invention follows the rule of increasing video memory efficiency and reducing computing efficiency, and sets the strategy priority queue as: Zero1>Zero2>Zero3.

[0031] Step 202: Build a distributed strategy cost evaluation database. When implementing a data parallel strategy, you must first build a model structure object in the CPU memory. For the case where the degree of parallelism is N, use fp32 precision to build N model objects in the CPU memory, and then load the model parameters from the hard disk to initialize the model object. Then upload the model structure to the graphics card after it is completed or split according to the distributed strategy. In this process, the free CPU memory and swap partition determine whether the model object can be built successfully, and the graphics card's video memory determines whether the model can be successfully transferred from the CPU memory to the video memory. Therefore, the present invention sets the CPU memory peak cost and the single card's video memory peak as policy cost evaluation indicators. For the above process, the CPU peak memory calculation formula is, Cost cpu-mem =(4P+R)*N, where P is the number of model parameters, N is the number of graphics cards, 4 corresponds to 4 bytes of fp32, and R is the system overhead other than the model state. In DeepSpeed, we found through experimental analysis that the value of R is 7. The calculation of peak memory overhead is determined according to different parallel strategies:

[0032] ZERO-1:12P / N+4P+2P*scale act

[0033] ZERO-2:14P / N+2P+2P*scale act

[0034] ZERO-3:16P / N+2P*scale act

[0035] Based on the above analysis, the peak video memory and peak memory overhead requirements of each policy in the policy priority queue are calculated.

[0036] Based on the above information, the following strategy key-value database list is designed:

[0037] {

[0038] policy:(ZERO-1, ZERO-2, ZERO-3)

[0039] level:(0,1,2)#The larger the number, the higher the priority

[0040] cpu-mem-peak:xxx.#The unit is G

[0041] gpu-mem-peak:xxx.#The unit is G

[0042] }

[0043] Step 3: Scan the existing computing resources, traverse the strategy priority queue, and select the first hit strategy as the recommended strategy.

[0044] Build an automatic idle resource scanning module. Mainly collect CPU available memory and swap partition space, idle graphics cards and their corresponding video memory and graphics card models. Build Deepspeed configuration template, calculate redundant resources, and give the best training hyperparameters. Details are as follows:

[0045] Step 301: Collect the idle computing power information of the current device. The computing resources related to the distributed training strategy include:

[0047] CPU memory, free swap storage, free graphics card memory and usage information. As for automatic scanning of free resources, we use the Python tool nvitop to read the free computing resources of the local machine. For multi-node situations, we use the paramiko tool to access remote nodes according to the configured hostfile to collect free computing resources.

[0048] Step 302: Traverse the priority queue, select the best strategy, and give a recommended configuration. According to the strategy priority queue and evaluation cost given in step 201 and step 202, start from the high-priority strategy and go to the low-priority strategy in turn. If the peak memory and peak video memory of the strategy that can be satisfied by the space resources are encountered, it is considered a hit strategy and output as a recommended strategy. In addition, a configuration template is built based on DeepSpeed, and the structure is as follows:

[0049]

[0050] Among them, STAGE_NUM points to the ZERO strategy type:

[0051]

[0052] MICRO_BATCH refers to the maximum batch size that a single model instance can support in a forward process. Define gpu_free_mem as the video memory of a single graphics card, gpu_mem_peak as the peak video memory overhead of the current strategy, L as the number of layers of Transformers in the current model, s as the maximum length of each sample, and h as the hidden layer feature dimension of the transformer output. The calculation method of MICRO_BATCH is as follows:

[0053] MICRO_BATCH=(gpu_free_mem-gpu_mem_peak) / 4Lhs.

[0054] The present invention can pre-calculate the overhead of video memory and CPU memory required for large model training; recommend appropriate distributed training strategies for currently available computing resources, and provide recommended training hyperparameters; a system using the above features can estimate the training overhead based on the selected large model type, scan node computing resources, provide training strategies and automatically build a configuration file for the training framework, and cooperate with the training platform to achieve efficient training of large models.

[0055] Beneficial effects: The present invention constructs an efficiency analysis system from the perspectives of computing efficiency and video memory efficiency. Starting from the priority of computing speed, combined with actual computing power, the proportion of video memory efficiency is continuously adjusted, the priorities of three data parallel strategies are given, and the cost calculation formula is given. A training strategy recommendation system is established that automatically selects parallel strategies and generates configuration files according to different computing power resource conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of the structure of a large model training distributed strategy recommendation system in an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the flow of a large model training distributed strategy recommendation system according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Figure 1 This is a structural diagram of the distributed strategy recommendation system for large model training in an embodiment of the present invention. The system structure includes a large model database 101, a strategy recommendation module 102, and a recommendation configuration output module 103; the large model database 101 is used to save the structure and model status information of common large models. Its construction method is shown in step 1.

[0060] The strategy recommendation module 102 is used to select the optimal strategy from the strategy priority queue according to the large model status information in the training request and the idle computing power status. According to the functional division, the module is divided into a distributed strategy priority queue 102-1, a computing power resource monitoring module 102-2 and a strategy decision unit 102-3. The distributed strategy priority queue 102-1 analyzes the computing efficiency and memory efficiency of the data parallel strategy, prioritizes the computing speed, and uses the space-for-time method to be compatible with low computing power to determine the priority of strategy selection. The computing power resource monitoring module 102-2 uses Prometheus to scan the status of each GPU of the cluster node, update the device status list in real time, and monitor the status of the idle computing power of the cluster node as an important basis for strategy selection. The strategy decision unit 102-3 is used to extract the optimal training strategy from the strategy priority queue according to the large model requirements and the idle computing power status. From ZERO-1, ZERO-2, and ZERO-3, first calculate the computing resources required for ZERO-1. If the current idle resources meet the requirements, ZERO-1 is used as the optimal training strategy, and so on.

[0061] The recommended configuration output module 103 is used to construct a configuration file based on DeepSpeed ​​according to the optimal strategy selected by the system, and calculate and output key configuration information such as batch size.

[0062] For large model training requests input by users, the following steps are required: Figure 2 The processing flow shown generates the configuration file of the optimal training strategy. The specific process is:

[0063] 201: extracting relevant model state information from the large model database according to the large model type to be trained selected by the user;

[0064] The large model database needs to be built in advance, mainly to save information related to the model name and model status. The collection of model information is mainly achieved by combing the model structure, including the model parameter quantity P, the ratio of the model's intermediate activation to the parameter quantity scale act , whether to use mixed precision training mark mixPrecision (default is True). Based on the above information, the model state overhead can be calculated: model parameters = model gradient = 2P, model optimizer = 12P, intermediate activation = scale act *P.

[0065] For example, the llama7B and llama13B models require different amounts of resources. Under the same computing resources, llama7B can use the Zero-1 training strategy, but llama13B can only use the Zero-3 training strategy. 202: Collect the status of idle computing power in the cluster, including idle CPU memory, idle swap storage, and idle graphics card model and memory capacity;

[0066] 203: Traverse the strategy priority queue in descending order, compare the matching degree between the training cost of each strategy on the current model and the actual idle computing power, and select the first hit strategy (or none of them are satisfied).

[0067] For example, when training the llama7B model, the current resources are 4 GPUs with 48G memory. According to the peak memory cost calculation formula, Zero-1 requires 49G memory, Zero-2 requires 38.5G memory, and Zero-3 requires 25.5G memory. The Zero-1 strategy is not satisfied, and Zero-2 is the first hit strategy. If the current resources are 4 GPUs with 80G memory, the Zero-1 strategy is satisfied, and Zero-1 is the first hit strategy.

[0068] When building a distributed training model, use free CPU memory + free swap as the threshold to determine whether multiple instances of the current strategy meet the initialization conditions. Based on the model state segmentation rules of the current strategy, calculate the video memory required for the model state on a single graphics card and compare it with the video memory of the free graphics card to determine whether the model can be successfully transferred from the CPU memory to the graphics card for startup.

[0069] 204: If there is a strategy output, the DeepSpeed ​​configuration file of the optimal strategy is given, and the hyperparameters (all hyperparameters) MICRO_BATCH in the configuration are further calculated and output. Otherwise, an empty file is output.

[0070] The above-mentioned examples are only preferred examples of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for evaluating large model training resources and recommending strategies, characterized in that: Step 1: Build a large model database to save model status data of different types and sizes; model types include LLaMA2, Baichuan2, QWen, ChatGLM2 and Mistral; model status includes: model parameter quantity, model gradient data quantity, model optimizer data quantity and model intermediate activation data quantity; model database supports expansion for synchronous update of the latest open source large models; Step 2: Build a computational cost evaluation database for different distributed strategies; define video memory efficiency as an indicator to measure the degree of redundant information stored on each graphics card; define computational efficiency as the ratio of the effective computation time of the graphics card, i.e., the time for model state update and the time for communication; generally, the higher the video memory efficiency, the lower the computational efficiency, which is suitable for scenarios with tight computing power; conversely, the lower the video memory efficiency, the higher the computational efficiency, which is suitable for scenarios with ample computing power; trigger from computational efficiency and video memory efficiency, analyze the video memory and communication overhead of the three stages of data parallel strategy, and design the strategy priority queue based on this; at the same time, from the perspective of engineering implementation, calculate the peak memory and peak video memory overhead of each strategy during training, and build a cost evaluation database; Step 3: Scan the existing computing resources, traverse the strategy priority queue, and select the first hit strategy as the recommended strategy; build an idle resource automatic scanning module; collect the CPU available memory and swap partition space, idle graphics cards and their corresponding video memory and graphics card models; build a Deepspeed configuration template, calculate redundant resources, and give the best training hyperparameters.

2. The method for large model training resource evaluation and strategy recommendation according to claim 1, characterized in that: The process of constructing the large model database specifically includes: collecting model information of LLaMA2, Baichuan2, QWen, ChatGLM2 and Mistral, and for each model, sorting out the model parameter quantities of the Embedding layer and the deEmbding layer in its structure, and recording them as P e and P de , and the number of parameters P of the transformer layer t , and the number of layers of transformers, then the overall number of parameters of the model is P = P e +P de +P t *Layers; at the same time, sort out the shape of the output tensor of each layer, and accumulate and calculate the amount of intermediate activation data P act； During the training process, each parameter corresponds to a gradient value, so the data volume of the model gradient is P; the model optimizer uses the Adam algorithm, in which each parameter corresponds to two variables, so the data volume of the optimizer is 2P; the current industry consensus is to use mixed precision training by default, that is, model parameters, intermediate activations and gradients use fp16 or BF16, and optimizer variables use fp32. In addition, the optimizer will also store a parameter backup of fp32 for parameter updates during training, so as to avoid errors caused by numerical overflow during mixed precision training as much as possible.

3. The method for large model training resource evaluation and strategy recommendation according to claim 1, characterized in that: The computational cost evaluation system for constructing different distributed strategies is specifically as follows: Step 201: Construct a distributed strategy priority queue based on the data parallel distributed algorithm ZERO; data parallelism is to run a complete model instance on each graphics card, but when the model is relatively large and the graphics memory of a single card cannot hold it, it is necessary to split the model state, with each card responsible for different parts, and use communication to simulate the training process of a single card running a single instance; ZERO has three strategies: ZERO-1, ZERO-2 and ZERO-3; assuming there are N graphics cards, each with M graphics memory; ZERO-1 splits the optimizer in the model state into N parts, with each card storing 1 / N of the optimizer variables corresponding to the parameters, ZERO-2 splits the optimizer and gradient data into N parts, and ZERO-3 splits the optimizer, gradient and model parameters into N parts; data parallel distributed strategies start from ZERO The evolution from -1 to ZERO-3 is to balance the video memory efficiency and computing efficiency. In ZERO-1, only the optimizer data is split, and the model instance in each graphics card saves the complete model parameters and gradient data. The proportion of its model state in the video memory is (12P / N+4P) / M. Similarly, ZERO-2 saves the complete model parameters, and the proportion of its model state in the video memory is (14P / N+2P) / M. ZERO-3 has a more complete split, and the proportion of its model state in the video memory is 16P / (N*M), without redundancy. From the perspective of training speed, the strategy with the highest computing efficiency is given priority. Combined with the actual limited idle computing power, the rule of increasing the video memory efficiency and reducing the computing efficiency is followed, and the strategy priority queue is set as: Zero1>Zero2>Zero3. Step 202: construct a distributed strategy cost evaluation database; when implementing a data parallel strategy, first construct a model structure object in the CPU memory; when the degree of parallelism is N, construct N model objects in the CPU memory using fp32 precision, and then load model parameters from the hard disk to initialize the model objects; then upload the model structure to the graphics card after it is completed or split according to the distributed strategy; In this process, the idle CPU memory and swap partition determine whether the model object can be built successfully, and the graphics card's video memory determines whether the model can be successfully transferred from the CPU memory to the video memory. Therefore, the CPU memory peak cost and the single card's video memory peak cost are set as the policy cost evaluation indicators. For the above process, the CPU peak memory calculation formula is: Cost cpu-mem =(4P+R)*N, where P is the number of model parameters, N is the number of graphics cards, 4 corresponds to 4 bytes of fp32, and R is the system overhead other than the model state. In DeepSpeed, the R value is 7. The peak memory overhead calculation is determined according to different parallel strategies: ZERO-1:12P / N+4P ZERO-2:14P / N+2P ZERO-3:16P / N Based on the above analysis, calculate the peak video memory and peak memory overhead requirements of each policy in the policy priority queue; based on the above information, design the following policy key-value database list: { policy:(ZERO-1, ZERO-2, ZERO-3) level:(0,1,2)#The larger the number, the higher the priority cpu-mem-peak:xxx.#The unit is G gpu-mem-peak:xxx.#The unit is G }。 4. The method for large model training resource evaluation and strategy recommendation according to claim 1, characterized in that: The construction process of scanning the existing computing resources, traversing the strategy priority table in order according to the priority of the distributed strategy, and selecting the first hit strategy as the recommended strategy is as follows: Step 301: Collect the idle computing power information of the current device; the computing resources related to the distributed training strategy include: CPU memory, free swap storage, free graphics card memory and usage information; for automatic scanning of free resources, the Python tool nvitop is used to read the free computing resources of the local machine; for multi-node situations, the paramiko tool is used to access remote nodes according to the configured hostfile to collect free computing resources; Step 302: Traverse the priority queue, select the best strategy, and give a recommended configuration; according to the strategy priority queue and evaluation cost given in step 301 and step 302, compare from the high priority strategy to the low priority strategy in turn. If the peak memory and peak video memory of the strategy that can be satisfied by the space resources are encountered, it is regarded as a hit strategy and output as a recommended strategy; in addition, build a configuration template based on DeepSpeed, the structure is as follows: Among them, STAGE_NUM points to the ZERO strategy type: MICRO_BATCH refers to the maximum batch size that a single model instance can support in a forward process. Define gpu_free_mem as the video memory of a single graphics card, gpu_mem_peak as the peak video memory overhead of the current strategy, L as the number of layers of Transformers in the current model, s as the maximum length of each sample, and h as the hidden feature dimension of the transformer output. The calculation method of MICRO_BATCH is as follows MICRO_BATCH=(gpu_free_mem-gpu_mem_peak) / 4Lhs.

5. The method for large model training resource evaluation and strategy recommendation according to claim 1, characterized in that: Step 1 specifically includes: collecting model information of LLaMA2, Baichuan2, QWen, ChatGLM2 and Mistral, and for each model, sorting out the model parameter quantities of the Embedding layer and deEmbding layer in its structure, and recording them as P e and P de , and the number of parameters P of the transformer layer t , and the number of layers of transformers, then the overall number of parameters of the model is P = P e +P de +P t * Layers; At the same time, the shape of the output tensor of each layer is sorted out, and the amount of intermediate activation data P is accumulated and calculated act During the training process, each parameter corresponds to a gradient value, so the amount of data for the model gradient is P.

6. The method for large model training resource evaluation and strategy recommendation according to claim 5, characterized in that: The model optimizer uses the Adam algorithm, in which each parameter corresponds to two variables, so the optimizer data volume is 2P. The current industry consensus is to use mixed precision training by default, that is, model parameters, intermediate activations and gradients use fp16 or BF16, and optimizer variables use fp32. In addition, the optimizer will also store a fp32 parameter backup for parameter updates during training, to avoid errors caused by numerical overflow during mixed precision training as much as possible.

7. A system for large model training resource evaluation and strategy recommendation according to any one of the methods of claims 1-6, characterized in that: It includes a large model database, a distributed training strategy database and a resource scanning and strategy recommendation module; the large model database is used to store model status information of common categories of large models; the distributed training strategy database is used to store the priority queues of commonly used distributed training strategies, and the basic training overhead of each strategy, and the training overhead is composed of peak video memory and peak memory; the resource scanning and strategy recommendation module is used to scan the idle computing resources in the existing system, and based on the distributed training strategy database, in the order of the priority queue, sequentially search for the best recommended training strategy and output the training configuration.

Images