Distributed training method and device of large model, electronic equipment and storage medium

Abstract

The invention provides a distributed training method and device of a large model, electronic equipment and a storage medium, and relates to the technical field of model training. The invention provides a gradient compression and expert network parallel collaborative cross-domain distributed training method. In the training process, gradient synchronization and expert parameter routing between the target expert sub-networks are controlled through a proposed collaborative compression and dynamic scheduling mechanism, the cross-domain communication traffic can be remarkably reduced, and the situation that the training efficiency is reduced due to insufficient network bandwidth is avoided. Meanwhile, on the premise that the model precision is kept stable, adaptive adjustment of the expert routing strategy and the gradient compression mode can be achieved, and the distributed training performance of the to-be-trained large model under the multi-cluster and heterogeneous computing power condition is improved. The method can effectively support a large-scale model training task in a cross-data center, cloud edge cooperation and computing power network environment, and can be called by a training platform, a scheduling system and a computing power service system as a bottom layer capability.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a distributed training method, apparatus, electronic device, and storage medium for large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale deep learning models, represented by Large Language Models (LLM), have experienced explosive growth in the number of parameters, expanding from billions to trillions. The computing power and storage resources of a single computer or a single computing cluster are far from sufficient to support the training needs of such models. Therefore, distributed training across regions and multiple clusters has become an indispensable key technology for large-scale model training. This training paradigm aims to aggregate the computing resources of data centers with different geographical locations and hardware architectures to jointly complete the training task of a massive model. However, network links in cross-domain environments typically have complex characteristics such as narrow bandwidth, high latency, large jitter, and dynamic changes. This makes frequent data exchanges between nodes a core bottleneck restricting overall training efficiency. At the same time, in order to further expand the model capacity without significantly increasing computational costs, the Mixture-of-Experts (MoE) model structure has been widely used. MoE achieves an efficient balance between parameter scale and computational efficiency by introducing multiple parallel "expert" sub-networks and dynamically selecting and activating a subset of experts for each input data by a gating network. However, in distributed training, the dynamic routing characteristics of the MoE model introduce additional challenges in expert parameter exchange and load balancing, further exacerbating the already severe cross-domain communication pressure. Therefore, how to efficiently and collaboratively train large-scale MoE models in such a complex heterogeneous environment has become a pressing technical challenge in this field.

[0003] Existing distributed training methods for large models utilize parameter server architectures or aggregated communication operations to uniformly manage and schedule parameter synchronization during training, aiming to shield the heterogeneity of underlying hardware and networks. These methods primarily focus on how to rationally divide and deploy a large number of experts across different computing devices, and design efficient routing algorithms and load balancing mechanisms to ensure that the computational load of each expert is as even as possible, avoiding some devices being idle while others are overloaded, thereby improving the utilization of computing resources.

[0004] However, the aforementioned distributed training methods for large models have high communication overhead and low model training efficiency. Summary of the Invention

[0005] This invention provides a distributed training method, apparatus, electronic device, and storage medium for large models, which addresses the low efficiency of distributed training of large models in the prior art and improves the efficiency of distributed training of large models.

[0006] This invention provides a distributed training method for large models, comprising: Based on the basic information of the large model to be trained input by the user, a distributed training scheme is determined; the distributed training scheme includes gradient compression and expert routing schemes; the large model to be trained includes the main model and multiple distributed expert subnetworks; The model body and at least one target expert subnetwork are trained in parallel based on an expert routing scheme. During parallel training, the gradient parameters of the model of each target expert subnetwork are compressed, transmitted, and summarized based on gradient compression to obtain compressed gradient information. The parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed.

[0007] According to the distributed training method for large models provided by this invention, the model gradient parameters of each target expert sub-network are compressed, transmitted, and summarized based on gradient compression to obtain compressed gradient information, including: Based on the gradient magnitude, sparsity, or variation range of the model gradient parameters, differential coding or quantization coding is dynamically selected as the gradient compression method. Based on the cross-domain link status, training phase, and load of each target expert subnetwork, the gradient compression ratio of the gradient compression method is determined. Based on the gradient compression ratio and gradient compression method, the model gradient parameters of each target expert subnetwork are compressed to obtain the gradient parameters of each target expert subnetwork. The gradient parameters of each target expert subnetwork are transmitted and summarized to obtain compressed gradient information.

[0008] According to the distributed training method for large models provided by this invention, a distributed training scheme is determined based on the basic information of the large model to be trained input by the user, including: Extract the model training classification dimension set from the basic information. The model training classification dimension set includes parameter size dimension, task complexity dimension, and data characteristic dimension. Cluster information is determined based on the node distribution, computing power level, and network bandwidth of the wide-area distributed cluster; Based on cluster information and the model training classification dimension set, a distributed training scheme is determined.

[0009] According to the distributed training method for large models provided by this invention, at least one target expert sub-network is trained in parallel based on an expert routing scheme, including: Based on the expert routing scheme, determine the sub-training schemes and the allocation of training resources for each target expert sub-network. The adjusted training resource allocation is obtained by adjusting the adjustment coefficient for each training resource allocation. Based on the adjusted training resource allocation and each sub-training scheme, each target expert sub-network is trained in parallel.

[0010] According to the distributed training method for large models provided by the present invention, the parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed, and the method further includes: The performance of the trained large model is tested based on the standard test dataset to obtain the performance metrics of the trained large model. Based on performance metrics and resource consumption metrics of the large model to be trained during the training process, the performance level of the large model after training is determined.

[0011] According to the distributed training method for large models provided by this invention, the parameters of the large model to be trained are iteratively updated based on compressed gradient information until the large model to be trained is completed, including: The compressed gradient information is fragmented and encrypted to obtain encrypted fragments; Select an adaptive transmission strategy based on the link bandwidth, latency, and load of the wide-area distributed cluster; Based on an adaptive transmission strategy, encrypted segments are transmitted to each target expert sub-network, so that the parameters of each target expert sub-network and the parameters of the main model are iteratively updated based on the encrypted segments until the large model to be trained is completed.

[0012] According to the distributed training method for large models provided by the present invention, the parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed, and the method further includes: Based on the performance level, application scenario requirements, and computing resource distribution of the trained large model, determine the deployment nodes and resource configuration scheme of the trained large model; Based on the deployment node and resource configuration scheme, the trained large model is distributed and deployed.

[0013] The present invention also provides a distributed training device for large models, comprising: The determination module is used to determine the distributed training scheme based on the basic information of the large model to be trained input by the user; the distributed training scheme includes gradient compression method and expert routing scheme; the large model to be trained includes the main model and multiple distributed expert subnetworks; The training module is used to perform parallel training of the main model and at least one target expert subnetwork based on an expert routing scheme. The compression module is used to compress, transmit, and summarize the model gradient parameters of each target expert sub-network based on gradient compression during parallel training, so as to obtain compressed gradient information. The update module is used to iteratively update the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a distributed training method for any of the large models described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a distributed training method for any of the large models described above.

[0016] This invention provides a distributed training method, apparatus, electronic device, and storage medium for large-scale models. Based on the basic information of the large-scale model to be trained input by the user, a distributed training scheme is determined. The distributed training scheme includes gradient compression and expert routing. The large-scale model to be trained includes a main model and multiple distributed expert sub-networks. The main model and at least one target expert sub-network are trained in parallel based on the expert routing scheme. During parallel training, the model gradient parameters of each target expert sub-network are compressed, transmitted, and summarized using gradient compression to obtain compressed gradient information. The parameters of the large-scale model to be trained are iteratively updated based on the compressed gradient information until the training is complete. To address the problems of high communication overhead, significant bandwidth bottlenecks, and complex parallel scheduling of expert networks in existing cross-domain distributed training methods, this invention proposes a cross-domain distributed training method that combines gradient compression and expert network parallelism. During training, gradient synchronization and expert parameter routing between target expert sub-networks are controlled through the proposed collaborative compression and dynamic scheduling mechanism, which significantly reduces cross-domain communication and avoids training efficiency degradation due to insufficient network bandwidth. Meanwhile, while maintaining stable model accuracy, this invention enables adaptive adjustments to expert routing strategies and gradient compression methods, improving the distributed training performance of large models under multi-cluster and heterogeneous computing power conditions. This invention effectively supports large-scale model training tasks across data centers, cloud-edge collaboration, and computing network environments, and can be used as a foundational capability by training platforms, scheduling systems, and computing service systems. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the distributed training method for large models provided by this invention.

[0019] Figure 2 This is the second flowchart of the distributed training method for large models provided by this invention.

[0020] Figure 3 This is a flowchart illustrating the process of determining a distributed training scheme provided by the present invention.

[0021] Figure 4 This is the third flowchart of the distributed training method for large models provided by this invention.

[0022] Figure 5 This is a flowchart illustrating the process of determining the performance level of a large model, as provided by the present invention.

[0023] Figure 6 This is the fourth flowchart of the distributed training method for large models provided by this invention.

[0024] Figure 7 This is a schematic diagram of the process for providing feedback to users on the training results of a large model, as provided by this invention.

[0025] Figure 8 This is a flowchart illustrating the process for determining hardware and environment optimization recommendations provided by the present invention.

[0026] Figure 9 This is a schematic diagram of the process for distributing and deploying a large, trained model, as provided by the present invention.

[0027] Figure 10 This is a schematic diagram of the process for monitoring the operation of a large model after it has been deployed, as provided by this invention.

[0028] Figure 11 This is a schematic diagram of the process for generating a runtime analysis report provided by the present invention.

[0029] Figure 12 This is a schematic diagram of the structure of the distributed training device for large models provided by the present invention.

[0030] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] The following is combined Figures 1-13 This invention describes a distributed training method, apparatus, and electronic device for large models.

[0033] Figure 1 This is one of the flowcharts illustrating the distributed training method for large models provided by this invention. Figure 2 This is the second flowchart illustrating the distributed training method for large models provided by this invention. Figure 1 and Figure 2 As shown, the distributed training method for large models includes steps S100 to S400, and the specific details of each step are as follows.

[0034] S100: Determine the distributed training scheme based on the basic information of the large model to be trained input by the user.

[0035] The distributed training scheme includes gradient compression and expert routing; the large model to be trained includes the main model and multiple distributed expert subnetworks.

[0036] The subject of this invention includes a cross-domain distributed training system (or system). The cross-domain distributed training system includes a gradient acquisition module, a gradient compression module, an expert selection and routing module, a cross-domain communication module, a training scheduling and control module, and a model update module.

[0037] This invention is applied to the training of large language models. The large model to be trained includes a distributed large language expert model (or application). The large model to be trained is a massive model, comprising multiple distributed expert subnetworks. Each expert subnetwork is distributed and stored across hundreds or thousands of GPUs, working collaboratively through gating networks and high-speed communication networks.

[0038] The expert routing scheme includes multiple target expert subnetworks that are matched with basic information input by the user.

[0039] Gradient compression refers to the method of compressing the model's gradient parameters during the training process of a distributed expert subnetwork. Compressing the model's gradient parameters can reduce the network communication burden and accelerate the training process.

[0040] like Figure 3As shown, users fill in and upload basic information about the large model to be trained through the interface of the cross-domain distributed training system. Users use interactive elements such as text boxes, dropdown menus, and file upload buttons on the system interface to fill in the basic information of the large model to be trained. Basic information includes model type (e.g., MoE), parameter size (10B, 100B, 200B, etc.), training objective task (natural language processing, knowledge graph construction, etc.), data source and scale, model description text (the system parses the model requirements based on the model description text), etc., and also upload application files such as the model structure file, initial parameter file, and training dataset file of the large model to be trained.

[0041] Optionally, if the user can only provide a vague description of their requirements, the cross-domain distributed training system can automatically analyze and generate a basic information framework for the large model to be trained through the large model requirement parsing module, combined with common industry training scenarios, guiding the user to supplement key details. Based on the basic information provided by the user or generated by the system parsing, key data crucial to the design of the distributed training solution, such as model parameter scale, data volume, and task type, are extracted, providing a basis for subsequent training configuration.

[0042] The cross-domain distributed training system interprets the user's input of basic information about the large model to be trained to determine the distributed training scheme. The distributed training scheme includes a gradient compression scheme and an expert routing scheme. The expert selection and routing module selects the corresponding target expert subnetwork based on the user's input and the load status of each node in the wide-area distributed cluster. The gradient compression module's dynamic compression control unit determines and adjusts the gradient compression method (e.g., gradient compression ratio) based on the cross-domain link status, training stage, and the load of each target expert subnetwork.

[0043] S200: The model body and at least one target expert subnetwork are trained in parallel based on an expert routing scheme.

[0044] The cross-domain distributed training system extracts at least one target expert subnetwork that needs to be trained from the expert routing scheme. All extracted target expert subnetworks, along with the main model, are trained in parallel. During parallel training, the computational resource requirements of each target expert subnetwork are allocated to the computing nodes of the wide-area distributed cluster.

[0045] During the execution of the distributed training scheme in the cross-domain distributed training system, the training process can be adaptively adjusted based on the cross-domain network conditions. After generating local gradients at each computing node, the cross-domain distributed training system first segments the gradients to reduce the data size of a single transmission. Then, based on the bandwidth, latency, and jitter of the cross-domain link, it dynamically adjusts the compression method and compression ratio of the segmented gradients. For models containing expert structures, the cross-domain distributed training system further selects and routes expert subnetworks based on the current input data characteristics and expert scores, realizing parallel expert training among multiple nodes.

[0046] S300: During parallel training, the gradient parameters of the model of each target expert sub-network are compressed, transmitted and summarized based on gradient compression to obtain compressed gradient information.

[0047] During parallel training, the model gradient parameters of each target expert subnetwork are collected. These parameters are then compressed using a gradient compression method to obtain compressed gradient information. Finally, the compressed gradient information is aggregated through communication links within the distributed training environment.

[0048] S400: Iteratively update the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed.

[0049] The compressed gradient information is decompressed to obtain the decoded model gradient parameters. Based on all the decoded model gradient parameters, the parameters of the main model and the parameters of each target expert sub-network are iteratively updated until the large model to be trained is completed.

[0050] Furthermore, during parallel training, information on the loss changes, gradient distribution changes, expert call frequency, and link bandwidth changes of the large model to be trained is collected in real time, and the gradient compression method and expert routing scheme are updated in real time based on this information.

[0051] Furthermore, the distributed training method for large models in this invention supports heterogeneous hardware platforms, including GPU clusters, AI acceleration chip clusters, edge node clusters, or hybrid computing environments across data centers.

[0052] This invention provides a distributed training method for large models. Based on the basic information of the large model to be trained input by the user, a distributed training scheme is determined. The distributed training scheme includes gradient compression and expert routing. The large model to be trained includes a main model and multiple distributed expert subnetworks. The main model and at least one target expert subnetwork are trained in parallel based on the expert routing scheme. During parallel training, the model gradient parameters of each target expert subnetwork are compressed, transmitted, and summarized using gradient compression to obtain compressed gradient information. The parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model training is complete. To address the problems of high communication overhead, prominent bandwidth bottlenecks, and complex parallel scheduling of expert networks (MoE) in existing cross-domain distributed training, this invention proposes a cross-domain distributed training method that combines gradient compression and expert network parallel collaboration. During training, gradient synchronization and expert parameter routing between target expert subnetworks are controlled through the proposed collaborative compression and dynamic scheduling mechanism, which can significantly reduce cross-domain communication and avoid training efficiency degradation due to insufficient network bandwidth. Meanwhile, while maintaining stable model accuracy, this invention enables adaptive adjustments to expert routing strategies and gradient compression methods, improving the distributed training performance of large models under multi-cluster and heterogeneous computing power conditions. This invention effectively supports large-scale model training tasks across data centers, cloud-edge collaboration, and computing network environments, and can be used as a foundational capability by training platforms, scheduling systems, and computing service systems.

[0053] Based on the above embodiments, a distributed training scheme is determined according to the basic information of the large model to be trained input by the user, including the following steps: Extract the model training classification dimension set from the basic information. The model training classification dimension set includes parameter size dimension, task complexity dimension, and data characteristic dimension. Cluster information is determined based on the node distribution, computing power level, and network bandwidth of the wide-area distributed cluster; Based on cluster information and the model training classification dimension set, a distributed training scheme is determined.

[0054] like Figure 4 As shown, a model training classification dimension set is extracted from the basic information. The model training classification dimension set includes parameter size dimension, task complexity dimension, and data characteristic dimension, so as to determine the training difficulty and resource requirement type of the large model to be trained based on the basic information.

[0055] The parameter scale dimension is determined based on the number of parameters in the large model to be trained. If the number of parameters in the large model to be trained is ≥150B, the parameter scale dimension is very large. If the number of parameters in the large model to be trained is ≤50B, the parameter scale dimension is small to medium. If the number of parameters in the large model to be trained is between 50B and 150B, the parameter scale dimension is large.

[0056] The task complexity dimension includes basic pre-training, specific fine-tuning, and multi-task collaborative training.

[0057] Data characteristics include structured data training, unstructured data training, and mixed data training.

[0058] Cluster information includes the node distribution, computing power level, network bandwidth, and other information of the wide-area distributed cluster.

[0059] Based on cluster information and the model training classification dimension set, suitable distributed training strategies, optimizer combinations, and communication synchronization mechanisms are selected to further determine the distributed training scheme. Distributed training strategies include model parallel training, data parallel training, and Mixture of Experts (MoE) model parallel training. For complex models spanning multiple classification dimensions (e.g., multi-task MoE models with extremely large parameters), the definition of a single classification label can be skipped, and multiple distributed training strategies can be directly adopted according to the characteristics of each module of the model. For example, the main body of the model can be trained in parallel, the expert branches can be trained in parallel using MoE models, and the training data can be trained in parallel using data.

[0060] Optionally, to improve training stability and the performance of the final trained large model, multiple training strategies can be combined for multiple rounds of training on the same large model to be trained. For example, the first round can use data parallel training for rapid iteration, the second round can use model parallel training to optimize deep parameters, and finally, MoE parallel training can be used to enhance task adaptability. The model result with the best performance in multiple rounds of training can be taken as the final training result, or the training parameters of multiple rounds can be weighted and fused.

[0061] Communication synchronization mechanisms include fully synchronous communication, semi-synchronous communication, and asynchronous communication. Optimizers combine inner AdamW with outer Nesterov momentum, and inner SGD with outer AdaGrad, etc.

[0062] This invention combines cluster information and model training classification dimension set to determine a distributed training scheme. It takes into account both the computing power of the wide-area distributed cluster and the various types of requirements in the training process of the large model to be trained, ensuring that the distributed training scheme matches both the computing power of the wide-area distributed cluster and the requirements of the large model to be trained.

[0063] Furthermore, a distributed training scheme is implemented. During the execution of the distributed training scheme, the training process is adaptively adjusted based on the cross-domain network conditions. After generating local gradients on each computing node, the cross-domain distributed training system first segments the gradients to reduce the data size of a single transmission. Subsequently, based on the bandwidth, latency, and jitter of the cross-domain link, the compression method and compression ratio of the segmented gradients are dynamically adjusted. For models containing expert structures, the cross-domain distributed training system further selects the target expert subnetwork and routes based on the features of the current input basic information and expert scores, realizing parallel expert training among multiple nodes. After completing gradient alignment, the cross-domain distributed training system uniformly updates the common parameters of the large model to be trained and the parameters of the target expert subnetwork. At the same time, it combines the link status and node load to schedule the training strategy in real time, thereby achieving a balance between communication overhead, training efficiency, and model accuracy, ensuring the stability and convergence performance of the cross-domain large model training process.

[0064] During the execution of the distributed training scheme, training progress and key metrics are monitored in real time. Monitoring metrics include the core metrics of the large model to be trained and the system operation metrics of the wide-area distributed cluster. Core metrics include loss value, accuracy, and convergence speed. System operation metrics include computing power utilization (CPU / GPU utilization), memory usage, network transmission rate, gradient transfer latency, etc.

[0065] Based on the above embodiments, parallel training of at least one target expert sub-network based on an expert routing scheme includes the following steps: Based on the expert routing scheme, determine the sub-training schemes and the allocation of training resources for each target expert sub-network. The adjusted training resource allocation is obtained by adjusting the adjustment coefficient for each training resource allocation. Based on the adjusted training resource allocation and each sub-training scheme, each target expert sub-network is trained in parallel.

[0066] The expert routing scheme identifies the target expert subnetworks that match the training data in the base dataset. Based on the characteristics of each target expert subnetwork, an independent sub-training scheme and training resource allocation are developed for each target expert subnetwork.

[0067] The training resource allocations for each subnetwork are adjusted using adjustment coefficients to obtain the adjusted allocations. Considering the additional communication overhead caused by data interaction and parameter passing between the target expert subnetworks, the training resource allocation for each subnetwork is multiplied by the adjustment coefficient to ensure additional redundancy in the training resource allocation during the training process, thereby ensuring overall training efficiency. For example, the adjustment coefficients range from 1.1 to 1.3.

[0068] Optionally, for large models to be trained with different requirements for training timeliness (such as rapid fine-tuning models for urgent business needs and deep pre-trained models for long-term technical research and development), different training resource scheduling priorities are adopted. Rapid fine-tuning models are given priority in allocating idle high-computing-power nodes and adopting a short-cycle synchronization strategy. Deep pre-trained models adopt balanced resource allocation and combine asynchronous communication to reduce resource waste and ensure continuous and stable training progress.

[0069] Based on the adjusted training resource allocation and each sub-training scheme, each target expert sub-network is trained in parallel.

[0070] This invention, by determining independent sub-training schemes for each target expert sub-network, ensures differentiated and targeted training based on the characteristics of each sub-network, thereby improving the accuracy of training. Furthermore, by adjusting the allocation of training resources through coefficients, this invention takes into account the additional communication overhead caused by data interaction and parameter transfer between target expert sub-networks during distributed training, thus improving the overall efficiency of training each sub-network.

[0071] Optional, such as Figure 6 As shown, if the large model to be trained is composed of multiple sub-models working together (for example, the large model to be trained is composed of a natural language understanding sub-model and a knowledge reasoning sub-model), then a sub-training scheme and a training resource allocation are specified for each sub-model. The training resource allocations are adjusted based on adjustment coefficients to obtain the adjusted training resource allocations. Based on the adjusted training resource allocations and the sub-training schemes, each sub-model is trained in parallel.

[0072] Based on the above embodiments, the gradient parameters of the model of each target expert sub-network are compressed, transmitted, and summarized using gradient compression to obtain compressed gradient information, including the following steps: Based on the gradient magnitude, sparsity, or variation range of the model gradient parameters, differential coding or quantization coding is dynamically selected as the gradient compression method. Based on the cross-domain link status, training phase, and load of each target expert subnetwork, the gradient compression ratio of the gradient compression method is determined. Based on the gradient compression ratio and gradient compression method, the model gradient parameters of each target expert subnetwork are compressed to obtain the gradient parameters of each target expert subnetwork. The gradient parameters of each target expert subnetwork are transmitted and summarized to obtain compressed gradient information.

[0073] Quantization coding is used to reduce the numerical precision of model gradient parameters by representing a gradient value with fewer bits. For example, standard model gradient parameters are typically represented as 32-bit floating-point numbers. Quantization coding maps these high-precision floating-point numbers to a finite, discrete set of low-precision numbers. Quantization coding converts FP32 (32 bits) to FP16 (16 bits), INT8 (8 bits), or INT4 (4 bits), which can theoretically result in compression of 2x, 4x, and 8x respectively, significantly reducing the amount of communication data.

[0074] Differential encoding does not directly transmit the complete current model gradient parameters; instead, it transmits the changes between the current and previously transmitted model gradient parameters. In successive training iterations, the changes in model gradient parameters typically do not change drastically. Therefore, the gradient difference between two iterations is often a sparse vector (with many values ​​close to zero) and a smaller numerical range. Differential encoding itself does not directly reduce the number of bits, but the difference vector it produces has better properties. Because the difference vector is sparser and smaller in value, it is very suitable for subsequent quantization or sparsification, thereby achieving a higher overall compression ratio.

[0075] The gradient acquisition module of the cross-domain distributed training system collects the model gradient parameters of each target expert subnetwork from the distributed storage nodes and generates the distribution information of model gradient parameters based on the granularity of the target expert subnetwork.

[0076] Furthermore, the gradient compression module includes a gradient segmentation encoding unit and a dynamic compression control unit. The gradient segmentation encoding unit dynamically selects differential encoding or quantization encoding as the gradient compression method based on the gradient magnitude, sparsity, or variation amplitude of the model gradient parameters. The dynamic compression control unit is used to adjust the gradient compression ratio of the gradient compression method according to the cross-domain link status of the cross-domain distributed training system, the training stage, and the load of each target expert sub-network.

[0077] The gradient segmented encoding unit compresses the model gradient parameters of each target expert subnetwork according to the gradient compression ratio and gradient compression method, thereby obtaining the gradient parameters of each target expert subnetwork.

[0078] The cross-domain communication module transmits and summarizes the gradient parameters of each target expert sub-network to obtain compressed gradient information.

[0079] This invention performs differential or quantized encoding on model gradient parameters based on compressed reference information, significantly reducing the communication load of the communication network in distributed training. While ensuring the training accuracy of the model, it reduces cross-domain communication bandwidth consumption and improves the efficiency of distributed training. Furthermore, this invention adjusts the gradient compression ratio in real time based on cross-domain link status, training phase, and the load of each target expert sub-network, achieving real-time optimization of the gradient compression method.

[0080] Based on the above embodiments, the parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed. The following steps are also included: The performance of the trained large model is tested based on the standard test dataset to obtain the performance metrics of the trained large model. Based on performance metrics and resource consumption metrics of the large model to be trained during the training process, the performance level of the large model after training is determined.

[0081] like Figure 5 As shown, the performance of a large trained model is tested based on a standard test dataset to obtain performance metrics. These metrics include accuracy, recall, convergence speed, and perplexity.

[0082] Based on performance metrics and resource consumption metrics of the large model under training, the performance level of the trained model is determined. Resource consumption includes computing time, network bandwidth consumption, and energy consumption. Resource consumption metrics include computing power utilization, GPU memory utilization, network transmission rate, and gradient transmission latency. The performance levels are categorized into four levels: Excellent, Good, Satisfactory, and Unsatisfactory.

[0083] like Figure 5 As shown, differential grading is determined based on the type of training task. The differential grading criteria are as follows.

[0084] For natural language processing training tasks (such as text classification and machine translation), the excellent standard is defined as a test set accuracy ≥ 95%, a perplexity value ≤ 3.5, and computational power utilization ≥ 85% during training. The good standard is an accuracy of 90%-95%, a perplexity value of 3.5-4.5, and computational power utilization of 80%-85%. The acceptable standard is an accuracy of 85%-90%, a perplexity value of 4.5-5.5, and computational power utilization of 75%-80%. Anything below the acceptable standard is considered unacceptable.

[0085] For knowledge graph construction training tasks, the excellent standard is defined as entity extraction accuracy ≥ 92%, relation extraction accuracy ≥ 90%, and no data transmission errors during the training period. The good standard is entity extraction accuracy 88%-92%, relation extraction accuracy 85%-90%, and data transmission error rate ≤ 0.5%. The acceptable standard is entity extraction accuracy 83%-88%, relation extraction accuracy 80%-85%, and data transmission error rate ≤ 1%. Anything below the acceptable standard is considered unacceptable.

[0086] For multi-task collaborative training tasks, the excellent standard is defined as an average accuracy of ≥93% for each subtask and a switching latency between tasks ≤100ms. The good standard is an average accuracy of 88%-93% for each subtask and a switching latency of 100-200ms between tasks. The acceptable standard is an average accuracy of 83%-88% for each subtask and a switching latency of 200-300ms between tasks. Anything below the acceptable standard is considered unacceptable.

[0087] For other types of training tasks (such as joint speech and semantic training), the excellent standard is defined as an overall performance score ≥ 94 points (out of 100) and a resource waste rate ≤ 5%. The good standard is an overall performance score of 89-94 points and a resource waste rate of 5%-10%. The qualified standard is an overall performance score of 84-89 points and a resource waste rate of 10%-15%. Anything below the qualified standard is considered unqualified.

[0088] Furthermore, for composite models, each sub-model is independently graded. The performance level of the composite model is comprehensively evaluated based on the synergistic effect between the sub-models. During the evaluation process, if a key sub-model fails, its training scheme is re-optimized and training is restarted.

[0089] Furthermore, the differentiated grading criteria are standardized and dynamically optimized. The thresholds for the differentiated grading criteria are updated by referencing industry benchmarks, historical training data, statistical analysis, and machine learning modeling.

[0090] This invention determines the performance level of a trained large model by using its performance and resource consumption metrics. It comprehensively considers the model's testing performance and resource consumption to evaluate the performance level of the trained large model, thus achieving an accurate evaluation of the performance level.

[0091] Furthermore, training results feedback and optimization suggestions are pushed out. For example... Figure 7 As shown, based on the performance level of the large model, the system evaluates the energy consumption of the application (the trained large model). The system then provides feedback on the training results of the large model to the user via a pop-up window. The training results include the performance level (including performance metrics and resource consumption metrics), optimization suggestions for performance consumption, and the option to deploy the model to a real-world application scenario (deployment performance selection).

[0092] The following optimization recommendations address performance consumption. For computationally intensive applications (e.g., deep learning training), the recommendation is to appropriately reduce the number of training iterations. For physical testing applications, the recommendation is to base optimization on I / O, computation count, cache hit rate, and network load. For composite applications, the recommendation is to provide independent assessments and separate optimization suggestions for each application.

[0093] Users input their decisions based on the training results of the large model displayed in the pop-up window. User decisions include direct deployment or rollback modifications.

[0094] Optionally, for large models with excellent performance, if users require further performance improvements, the cross-domain distributed training system can provide advanced optimization suggestions, such as increasing training data diversity and fine-tuning optimizer parameters, based on the structural characteristics of the large model. For large models with good or acceptable performance, optimization suggestions are provided for addressing performance shortcomings (such as insufficient accuracy or slow convergence speed), including adjusting the number of model layers, optimizing the data preprocessing process, and changing the parallel training strategy. For large models with unacceptable performance, the reasons for training failure are analyzed (such as unreasonable parameter initialization, poor data quality, or insufficient resource allocation), and targeted solutions are provided.

[0095] Optional, such as Figure 8 As shown, for large models trained using physical node clusters, key data during the training process is collected. The cross-domain distributed training system can identify resource bottleneck nodes based on data such as node computing power fluctuations, network latency changes, and peak memory usage recorded during training, and provide hardware and environment optimization suggestions such as upgrading node computing power, expanding network bandwidth, and optimizing memory configuration.

[0096] Optionally, for composite models consisting of multiple sub-models, the system provides feedback on the training results (performance metrics and resource consumption metrics) of each sub-model, and offers independent optimization suggestions for each sub-model based on its weaknesses. At the same time, it analyzes the collaboration issues between sub-models and provides optimization solutions for the interaction process.

[0097] For models that need optimization, users can choose to retrain directly based on the existing solution, or modify the model parameters, data, or training configuration according to the optimization suggestions and then re-upload for training. Alternatively, they can choose not to train for the time being and save the current results to the history library.

[0098] Based on the above embodiments, the parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed, including the following steps: The compressed gradient information is fragmented and encrypted to obtain encrypted fragments; Select an adaptive transmission strategy based on the link bandwidth, latency, and load of the wide-area distributed cluster; Based on an adaptive transmission strategy, encrypted segments are transmitted to each target expert sub-network, and each target expert sub-network is iteratively updated based on the encrypted segments until the large model to be trained is completed.

[0099] To improve the transmission efficiency of compressed gradient information, the gradient compression module of the cross-domain distributed training system segments and encrypts the compressed gradient information to obtain encrypted segments.

[0100] The cross-domain communication module of the cross-domain distributed training system selects an adaptive transmission strategy based on the link bandwidth, latency, and load of the wide-area distributed cluster.

[0101] The cross-domain communication module transmits encrypted fragments to each target expert sub-network according to an adaptive transmission strategy, so as to iteratively update each target expert sub-network and the main body of the large model to be trained based on the encrypted fragments until the large model to be trained is completed.

[0102] This invention selects an adaptive transmission strategy based on link bandwidth, latency, and load conditions, which can optimize the broadcasting of compressed gradient information in real time, improve the communication efficiency between various target expert sub-networks, and help improve the parameter update efficiency and training efficiency of the large model to be trained.

[0103] Furthermore, for large models requiring optimization, users adjust the model's structure (e.g., adding or removing network layers, modifying the attention mechanism), optimize training data (e.g., cleaning noisy data, supplementing scarce samples), and adjust training parameters (e.g., learning rate, batch size) based on the system's optimization suggestions, and then re-upload the optimized model and data files. Upon receiving the files, the system automatically matches the optimized distributed training scheme, initiates the retraining process, and provides real-time feedback on the training progress and evaluation information, allowing users to decide whether to continue optimization or deployment.

[0104] Based on the above embodiments, the parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed. The following steps are also included: Based on the performance level, application scenario requirements, and computing resource distribution of the trained large model, determine the deployment nodes and resource configuration scheme of the trained large model; Based on the deployment node and resource configuration scheme, the trained large model is distributed and deployed.

[0105] like Figure 9 As shown, firstly, based on the model's hardware requirements (such as GPU model, memory size, storage capacity, etc.) and software dependencies (such as operating system version, deep learning framework version, etc.), candidate deployment nodes and candidate deployment resources that meet the basic conditions are selected.

[0106] Secondly, deployment nodes are matched according to the characteristics of the application scenario (application scenario requirements). For example, real-time interactive applications (e.g., intelligent customer service, voice assistant, etc.) are prioritized for deployment to edge nodes closer to users; batch data processing applications (e.g., large-scale text analysis, data mining, etc.) are deployed to data center clusters; and applications with high computing power requirements (e.g., complex reasoning, large sample generation) are deployed to supercomputing or intelligent computing centers.

[0107] Finally, by combining the energy consumption characteristics of the model with the energy cost of the deployment nodes (determined by the performance level), the candidate deployment resources are optimized. For example, low-energy models are deployed to high-cost nodes in the city center to ensure response speed, while high-energy models are deployed to low-cost nodes in the suburbs to reduce operating costs while meeting performance requirements.

[0108] Optionally, before model deployment, the system simulates and calculates key indicators such as inference latency, throughput, and energy consumption per unit time of the model on the target deployment node based on the hardware parameters (such as GPU computing power and network bandwidth) and model characteristics of the target deployment node, and provides feedback to the user on the expected running effect after deployment, so that the user can confirm whether to execute the deployment.

[0109] This invention determines the deployment nodes and resource configuration scheme of a trained large model by considering its performance level, application scenario requirements, and computing resource distribution. It fully considers various factors in the deployment process of the trained large model and can achieve accurate deployment of the trained large model.

[0110] Furthermore, such as Figure 10 As shown, after the trained large model is deployed to the target computing power node, a runtime monitoring module is created. The runtime monitoring module is used to collect and store multi-dimensional data in real time, including the running status of the large model, real-time resource consumption, service quality indicators, runtime lifecycle events, and application scenario-related data.

[0111] Real-time resource consumption includes GPU / CPU utilization, memory / video memory usage, network input / output rate, and energy consumption per unit time. This data is collected through the hardware monitoring interface of the deployment node and system software logs, while also recording fluctuations and peak data in resource consumption.

[0112] Service quality metrics include inference response latency (e.g., average latency, maximum latency, 95th percentile latency), request throughput (e.g., number of requests processed per unit time), service success rate (e.g., percentage of requests successfully responded to), error types and frequency (e.g., inference errors, timeout errors), and service quality variations under different loads (e.g., low load, medium load, high load).

[0113] The runtime lifecycle events include model start / stop events, resource scaling events (e.g., expanding the number of GPUs from 2 to 4), model version update events, and node failure and migration events. The time, cause, and impact on the service of each event are recorded, and the additional resource consumption and service interruption duration caused by the events are statistically analyzed.

[0114] Application scenario-related data includes user distribution of model services (including geographic and industry distribution), request sources (e.g., web, mobile, API calls), and business data (e.g., dialogue rounds and problem resolution rates in intelligent customer service scenarios). Analyzing the relationship between these data and the operational status of the large model provides business perspective support for subsequent optimization.

[0115] Furthermore, such as Figure 11 As shown, after monitoring the operation of the trained large model, the operational data is analyzed, and the analysis includes: classifying the trained large model; for one-time operation tasks (such as processing a specific batch of data), data analysis is performed after the task is completed, and an operation analysis report is generated; for continuous service models, the operational data of the trained large model is comprehensively analyzed according to a preset period (e.g., daily, weekly), and periodic operation reports are generated.

[0116] Perform basic statistical analysis on the trained large model (hereinafter referred to as the large model): Statistical analysis is performed on data such as resource consumption, service quality, and lifecycle events during the operation of the large model. Statistical indicators such as average, maximum, minimum, and fluctuation range are calculated and presented in the form of charts (line charts, bar charts, heat maps) to intuitively show the model's operating trend.

[0117] Optionally, if abnormal energy consumption (such as a sudden increase of more than 50% in energy consumption per unit time) or a decline in service quality (such as a response delay exceeding 10 times the threshold) is detected during model operation, the system will initiate anomaly diagnosis. By correlating and analyzing resource consumption data, event logs, and service request data, the system will locate the cause of the anomaly (such as hardware failure, a sudden increase in request volume, or abnormal model parameters) and provide a detailed description of the anomaly, its root cause, and its scope of impact in the report.

[0118] Optionally, based on the analysis of the causes of anomalies, the system provides specific solution suggestions for different types of anomalies, such as recommending to replace nodes for hardware failures, suggesting to start elastic scaling for sudden increases in request volume, and suggesting to roll back to a historical stable version for abnormal large model parameters.

[0119] Optionally, for large models that run continuously, if the resource utilization of the currently deployed node is detected to be low for a long period of time (e.g., GPU utilization is consistently below 30%) or the energy consumption cost is too high, the system combines the resource status of other candidate nodes with the migration cost to calculate the expected benefits after migration (such as the cost reduction ratio and performance improvement), and provides users with migration suggestions for large models to help optimize resource allocation.

[0120] Optionally, a full lifecycle optimization module for large models can be built. After the large model is trained, deployed, and run for a period of time, the distributed training scheme, model structure, and deployment strategy can be continuously optimized by combining historical training data, operation monitoring data, and application scenario feedback. At the same time, the performance grading standards and resource scheduling algorithms of large models can be improved.

[0121] Furthermore, the distributed training scheme is optimized by analyzing the relationship between training strategies and the performance of large models under different large models and different amounts of data. The combination of model parallel training, data parallel training and MoE parallel training is adjusted, and optimizer parameters (such as learning rate decay strategy and momentum coefficient) are optimized to improve training efficiency and model convergence speed.

[0122] Furthermore, we will improve the performance level classification criteria for large models, and update the performance indicator thresholds for various types of tasks by combining new training cases and industry technology development trends (for example, with technological advancements, we will raise the excellent accuracy standard for natural language processing tasks from 95% to 97%), and add new performance evaluation dimensions (such as model interpretability and robustness).

[0123] Furthermore, the resource scheduling algorithm is optimized by analyzing historical deployment data through machine learning to establish a mapping relationship between model characteristics, application scenarios, and deployment node performance. This improves the accuracy of resource selection and matching, reduces resource waste, and achieves dynamic load balancing scheduling by combining real-time resource load conditions.

[0124] Furthermore, optimize the model operation optimization strategy. Based on the resource consumption and service quality data during model operation, adjust the model's inference optimization parameters (such as quantization accuracy and pruning ratio) to reduce operating energy consumption and resource usage while ensuring performance.

[0125] Furthermore, the relationship between the input of model training (e.g., resource consumption, time cost) and the application benefits (e.g., improved business efficiency, revenue growth) is analyzed to establish an input-output ratio evaluation model, providing users with decision-making references for training and deployment, and helping users balance technology input and business benefits.

[0126] This invention, based on traditional distributed training methods, proposes a parallel collaborative mechanism for gradient compression and expert networks (MoE) in cross-domain environments. This mechanism includes gradient importance awareness, expert load scheduling, dynamic compression control, and cross-domain communication optimization, and can be provided as a basic training capability to upper-layer training platforms, scheduling systems, and computing power collaborative service systems. This invention offers the following beneficial technical effects: 1. The proposed training process covers key aspects of model training in cross-data center and multi-cluster heterogeneous computing environments, including gradient synchronization, expert routing, and cross-domain node communication, and is applicable to various training architectures. 2. During model iterative training, this invention can dynamically adjust the gradient compression rate and expert parameter transmission path based on network link status and expert load, thereby maintaining stable training efficiency under different communication conditions. 3. By co-optimizing the gradient compression mechanism with the expert selection strategy, this invention achieves a balance between training communication volume, model accuracy, and convergence speed, forming an adaptive closed-loop tuning process for training.

[0127] The distributed training device for large models provided by the present invention will be described below. The distributed training device for large models described below can be referred to in correspondence with the distributed training method for large models described above.

[0128] like Figure 12 As shown, a distributed training device for a large model includes: The determination module 501 is used to determine the distributed training scheme based on the basic information of the large model to be trained input by the user; the distributed training scheme includes gradient compression method and expert routing scheme; the large model to be trained includes the main model and multiple distributed expert subnetworks. Training module 502 is used to perform parallel training of the main model and at least one target expert subnetwork based on an expert routing scheme. The compression module 503 is used to compress, transmit, and summarize the model gradient parameters of each target expert sub-network based on gradient compression during parallel training to obtain compressed gradient information. The update module 504 is used to iteratively update the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed.

[0129] The distributed training device for large models provided by this invention determines a distributed training scheme based on the basic information of the large model to be trained input by the user. The distributed training scheme includes a gradient compression method and an expert routing scheme. The large model to be trained includes a main model and multiple distributed expert subnetworks. The main model and at least one target expert subnetwork are trained in parallel based on the expert routing scheme. During the parallel training process, the model gradient parameters of each target expert subnetwork are compressed, transmitted, and summarized based on the gradient compression method to obtain compressed gradient information. The parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed. To solve the problems of high communication overhead, prominent bandwidth bottlenecks, and complex parallel scheduling of expert networks (MoE) in existing cross-domain distributed training, this invention proposes a cross-domain distributed training method that combines gradient compression and expert network parallel collaboration. During the training process, gradient synchronization and expert parameter routing between target expert subnetworks are controlled by the proposed collaborative compression and dynamic scheduling mechanism, which can significantly reduce cross-domain communication volume and avoid training efficiency degradation due to insufficient network bandwidth. Meanwhile, while maintaining stable model accuracy, this invention enables adaptive adjustments to expert routing strategies and gradient compression methods, improving the distributed training performance of large models under multi-cluster and heterogeneous computing power conditions. This invention effectively supports large-scale model training tasks across data centers, cloud-edge collaboration, and computing network environments, and can be used as a foundational capability by training platforms, scheduling systems, and computing service systems.

[0130] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0131] Figure 13 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 13As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a distributed training method for a large model. This method includes: determining a distributed training scheme based on the basic information of the large model to be trained input by the user; the distributed training scheme includes a gradient compression method and an expert routing scheme; the large model to be trained includes a model body and multiple distributed expert subnetworks; parallel training of the model body and at least one target expert subnetwork based on the expert routing scheme; during the parallel training process, compressing, transmitting, and summarizing the model gradient parameters of each target expert subnetwork based on the gradient compression method to obtain compressed gradient information; iteratively updating the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is trained.

[0132] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the distributed training method for large models provided by the above methods. The method includes: determining a distributed training scheme based on the basic information of the large model to be trained input by the user; the distributed training scheme includes a gradient compression method and an expert routing scheme; the large model to be trained includes a model body and multiple distributed expert subnetworks; performing parallel training on the model body and at least one target expert subnetwork based on the expert routing scheme; during the parallel training process, compressing, transmitting, and summarizing the model gradient parameters of each target expert subnetwork based on the gradient compression method to obtain compressed gradient information; iteratively updating the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is trained.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed training method for large models, characterized in that, include: Based on the basic information of the large model to be trained input by the user, determine the distributed training scheme; The distributed training scheme includes gradient compression and expert routing schemes; The large model to be trained includes the main model body and multiple distributed expert sub-networks; The model body and at least one target expert subnetwork are trained in parallel based on the expert routing scheme. During parallel training, the model gradient parameters of each of the target expert sub-networks are compressed, transmitted, and summarized based on the gradient compression method to obtain compressed gradient information. The parameters of the large model to be trained are iteratively updated based on the compressed gradient information until the large model to be trained is completed.

2. The distributed training method for large models according to claim 1, characterized in that, The step of compressing, transmitting, and summarizing the model gradient parameters of each of the target expert sub-networks based on the gradient compression method to obtain compressed gradient information includes: Based on the gradient magnitude, sparsity, or variation range of the model gradient parameters, differential encoding or quantization encoding is dynamically selected as the gradient compression method. The gradient compression ratio of the gradient compression method is determined based on the cross-domain link status, the training phase, and the load of each target expert sub-network. Based on the gradient compression ratio and the gradient compression method, the model gradient parameters of each of the target expert sub-networks are compressed to obtain the gradient parameters of each of the target expert sub-networks. The gradient parameters of each of the target expert sub-networks are transmitted and summarized to obtain the compressed gradient information.

3. The distributed training method for large models according to claim 1, characterized in that, The step of determining a distributed training scheme based on the basic information of the large model to be trained input by the user includes: The model training classification dimension set is extracted from the basic information. The model training classification dimension set includes parameter size dimension, task complexity dimension, and data characteristic dimension. Cluster information is determined based on the node distribution, computing power level, and network bandwidth of the wide-area distributed cluster; Based on the cluster information and the model training classification dimension set, the distributed training scheme is determined.

4. The distributed training method for large models according to claim 1, characterized in that, The parallel training of at least one target expert sub-network based on the expert routing scheme includes: Based on the expert routing scheme, determine each sub-training scheme and each training resource allocation for each of the target expert sub-networks; The adjusted training resource allocation is obtained by adjusting the adjustment coefficient for each of the above-mentioned training resource allocations. Based on the adjusted training resource allocation and the sub-training schemes, the target expert sub-networks are trained in parallel.

5. The distributed training method for large models according to claim 1, characterized in that, The step of iteratively updating the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed also includes: The performance of the trained large model is tested based on a standard test dataset to obtain the performance metrics of the trained large model. Based on the performance metrics and the resource consumption metrics of the large model to be trained during the training process, the performance level of the large model after training is determined.

6. The distributed training method for large models according to claim 1, characterized in that, The iterative update of the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed includes: The compressed gradient information is fragmented and encrypted to obtain encrypted fragments; Select an adaptive transmission strategy based on the link bandwidth, latency, and load of the wide-area distributed cluster; Based on the adaptive transmission strategy, the encrypted segments are transmitted to each of the target expert sub-networks, so as to iteratively update the parameters of each target expert sub-network and the parameters of the model body based on the encrypted segments, until the large model to be trained is completed.

7. The distributed training method for large models according to claim 1, characterized in that, The step of iteratively updating the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed also includes: Based on the performance level, application scenario requirements, and computing resource distribution of the trained large model, the deployment nodes and resource configuration scheme of the trained large model are determined. Based on the deployment nodes and the resource configuration scheme, the trained large model is deployed.

8. A distributed training device for a large model, characterized in that, include: The determination module is used to determine the distributed training scheme based on the basic information of the large model to be trained input by the user; The distributed training scheme includes gradient compression and expert routing; the large model to be trained includes the main model and multiple distributed expert subnetworks. The training module is used to perform parallel training on the main model and at least one target expert subnetwork based on the expert routing scheme. The compression module is used to compress, transmit, and summarize the model gradient parameters of each of the target expert sub-networks based on the gradient compression method during parallel training, so as to obtain compressed gradient information. The update module is used to iteratively update the parameters of the large model to be trained based on the compressed gradient information until the large model to be trained is completed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the distributed training method for the large model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed training method for the large model as described in any one of claims 1 to 7.

Images