A Distributed Training Method and System for Large-Scale Electric Power Models Based on FSDP and Gradient Compression

Abstract

This invention discloses a distributed training method and system for a large-scale power model based on FSDP and gradient compression. The method includes: S1, constructing a multimodal power dataset; S2, partitioning the dataset according to power scenarios and equipment types; S3, constructing a hybrid parallel training framework; S4, prioritizing the parallel execution of computationally intensive and communication-intensive tasks, using the NCCL library to implement asynchronous gradient partitioning transmission; S5, executing distributed training, utilizing GPU cluster computing power to accelerate model convergence; S6, real-time monitoring of training speed and model accuracy metrics, determining whether the targets are met, and if so, outputting the accelerated training model; otherwise, dynamically adjusting the gradient compression rate and communication strategy according to the training stage and executing step S3. This invention introduces a hybrid parallel framework combining FSDP and power feature-aware gradient compression, supplemented by scenario-based data partitioning and a dynamic feedback mechanism, significantly reducing GPU memory usage, improving training speed, and ensuring model accuracy.

Description

Technical Field

[0001] This invention relates to a distributed training method for a large power model in the field of power systems, and more particularly to a distributed training method for a large power model based on FSDP and gradient compression, and also to a distributed training system for a large power model based on FSDP and gradient compression. Background Technology

[0002] With the intelligent development of the power industry, the parameter scale of large-scale multimodal power models based on deep learning has reached the billion level, posing a severe challenge to their training process. Existing mainstream data-parallel (DP) training methods require storing a complete copy of the model on each GPU, resulting in a single GPU memory usage exceeding 65GB on hardware such as the NVIDIA A100 (80GB VRAM), which cannot accommodate the complete model, leading to frequent training interruptions and hindering effective implementation. Simultaneously, the massive model parameters and gradients cause a surge in cross-GPU communication, requiring the transfer of hundreds of GB of data per iteration in an 8-GPU cluster, with communication time accounting for over 40%, resulting in lengthy model training cycles and severely restricting the research, development, and application iteration efficiency of large-scale power models.

[0003] To alleviate these problems, the industry has attempted to use gradient compression techniques, such as gradient discarding or quantization, to reduce the amount of communication data. However, these general compression methods are not optimized for the unique characteristics of power data. Power data generally exhibits characteristics such as sparse equipment defect samples and high dimensionality of technical terminology features. General, crude compression can introduce significant gradient errors, leading to difficulties in model convergence and a significant decrease in accuracy for power-specific tasks (such as defect detection and terminology recognition). The mean accuracy (mAP) may decrease by 3-5%, failing to meet the accuracy requirements of power scenarios while ensuring training efficiency. Summary of the Invention

[0004] To address the technical problems of excessive memory consumption and slow training speed in existing large-scale power model training methods, this invention provides a distributed training method and system for large-scale power models based on FSDP and gradient compression.

[0005] This invention employs the following technical solution: a distributed training method for a large-scale power model based on FSDP (Fully Sharded Data Parallel) and gradient compression, comprising the following steps: S1. Construct a multimodal power dataset; S2. Divide the dataset into segments according to power scenarios and equipment types; S3. Construct a hybrid parallel training framework, which includes: splitting the large power model into visual features and linguistic features according to visual and linguistic branches; storing model parameters, gradients, and optimizer states in multiple GPUs using a fully sharded data parallel approach; compressing the visual feature gradients using low-rank decomposition while retaining principal components; compressing the linguistic feature gradients using non-uniform quantization, and ensuring that the key term errors for the power term embedding gradients do not exceed a preset error. S4. Prioritize the parallel execution of computationally intensive and communication-intensive tasks, and use the NCCL library to implement gradient fragmentation and asynchronous transmission; S5. Perform distributed training and use the computing power of GPU clusters to accelerate model convergence; S6. Monitor the training speed and model accuracy in real time and determine whether they meet the standards. If they do, output the accelerated training model. Otherwise, dynamically adjust the gradient compression rate and communication strategy according to the training stage and execute step S3.

[0006] This invention introduces a hybrid parallel framework combining fully fragmented data parallelism (FSDP) with power feature-aware gradient compression, supplemented by scenario-based data fragmentation and dynamic feedback mechanisms. This reduces the single-card memory usage to below 30GB, optimizes cross-card communication strategies, reduces communication volume by combining gradient compression, and designs a non-uniform compression algorithm for power data features to ensure that the model's mAP index drops to below 1% after compression. This solves the technical problems of excessive memory usage and slow training speed in existing large power model training methods, achieving the technical effect of significantly reducing memory usage, improving training speed, and ensuring model accuracy.

[0007] As a further improvement to the above scheme, the vision branch is based on the Swing Transformer architecture, and the language branch is based on the BERT architecture.

[0008] As a further improvement to the above scheme, in step S3, the compression ratio of the low-rank decomposition is 4:1, and the first 25% of the principal components are retained. The gradient quantization is 8 bits, and the preset error is 2%.

[0009] As a further improvement to the above scheme, in step S6, compression is turned off in the early stage of training, turned on in the middle stage, and reduced to 2:1 in the later stage.

[0010] As a further improvement to the above scheme, the dataset includes at least 5 million images of substation inspections and equipment close-ups, and at least 2 million terminology texts. The images are preprocessed by scene into 512×512 pixel RGB format.

[0011] As a further improvement to the above scheme, in step S2, three subsets are divided based on the defect labels of power station equipment, power distribution lines and transformer substations, and each GPU node is assigned an independent scene subset to adapt the data to the model branches.

[0012] As a further improvement to the above scheme, in step S3, the visual features are compressed using the singular value decomposition method, the language feature gradient is compressed using 8-bit non-uniform quantization, and the dynamic switch is adjusted according to the training stage.

[0013] As a further improvement to the above scheme, in step S6, when the training speed improvement does not reach 20%, the gradient compression ratio is increased; when the model accuracy index mAP decreases by more than 1%, the gradient compression ratio is decreased or the model branch parameter ratio is adjusted.

[0014] Furthermore, the parameters of the visual branch and the language branch account for 60% and 40% respectively, and the two branches are deployed on different nodes of multiple GPU clusters.

[0015] This invention also provides a distributed training system for a large-scale power model based on FSDP and gradient compression, which applies any of the above-described distributed training methods for large-scale power models based on FSDP and gradient compression; the system includes: The dataset building module is used to build a multimodal power dataset. A data sharding module is used to shard the dataset according to power scenarios and equipment types; The framework construction module is used to build a hybrid parallel training framework, and includes a model splitting unit, a parameter sharding unit, and a gradient compression unit. The model splitting unit is used to split the large power model into visual features and linguistic features according to visual and linguistic branches. The parameter sharding unit is used to store model parameters, gradients, and optimizer states in shards on multiple GPUs using a fully sharded data parallel approach. The gradient compression unit is used to compress the gradients of the visual features using low-rank decomposition while retaining principal components. The gradient compression unit is also used to compress the gradients of the linguistic features using non-uniform quantization, and for the embedding gradients of power terms, it retains the key term error to be no more than a preset error. The communication optimization module is used to prioritize the parallel execution of computationally intensive and communication-intensive tasks, and uses the NCCL library to implement gradient fragmentation asynchronous transmission. The training module is used to perform distributed training, leveraging the computing power of GPU clusters to accelerate model convergence. The evaluation feedback adjustment module is used to monitor training speed and model accuracy metrics in real time and determine whether they meet the standards. If they do, it outputs a training acceleration model; otherwise, it dynamically adjusts the gradient compression rate and communication strategy according to the training phase and runs the framework construction module.

[0016] Compared with existing large-scale power model training methods, the distributed training method and system for large-scale power models based on FSDP and gradient compression of the present invention has the following advantages: 1. This distributed training method for large-scale power models based on FSDP and gradient compression introduces a hybrid parallel framework combining fully sharded data parallelism (FSDP) with power feature-aware gradient compression. It also incorporates scenario-based data sharding and dynamic feedback mechanisms to reduce single-card memory usage to below 30GB. Furthermore, it optimizes cross-card communication strategies, reduces communication volume through gradient compression, and designs a non-uniform compression algorithm tailored to power data characteristics to ensure that the model's mAP (memory accuracy) drops to a low level after compression. This solves the technical problems of excessive memory usage and slow training speed in existing large-scale power model training methods, achieving significant reductions in memory usage, improved training speed, and guaranteed model accuracy.

[0017] 2. This distributed training method for large-scale power models based on FSDP and gradient compression can reduce the single-card memory usage from 65GB to 25GB, supporting stable training of a 1 billion-parameter power model on an 8-card A100 cluster. Furthermore, the training cycle on the 8-card A100 cluster can be shortened from 14 days to 11.2 days, a 20% speed improvement. In addition, accuracy loss is controllable; gradient compression reduces the mAP by ≤1%, scenario-based data sharding improves defect detection recall by 3%, and non-uniform gradient compression protects terminology feature gradients, maintaining terminology recognition accuracy above 92%.

[0018] 3. The distributed training system for the large power model based on FSDP and gradient compression has the same beneficial effects as the training methods mentioned above, and will not be elaborated here. Attached Figure Description

[0019] Figure 1 This is a flowchart of the distributed training method for a large power model based on FSDP and gradient compression, as described in Embodiment 1 of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Example 1 Please see Figure 1This embodiment provides a distributed training method for a large-scale power model based on FSDP (Fully Sharded Data Parallel) and gradient compression. This method primarily accelerates the training of the large-scale power model, outputs an accelerated training model, and solves problems such as high GPU memory usage, slow training speed, and accuracy loss due to gradient compression in existing technologies. It should be noted that this method is compatible with GPU cluster environments such as NVIDIA A100, and in other embodiments, it can also be adapted to other GPU cluster environments as needed. The method includes steps S1-S6; of course, in other embodiments, other steps may be included according to actual requirements.

[0022] S1. Construct a multimodal power dataset. In this embodiment, the dataset includes at least 5 million images of substation inspections and equipment close-ups, and at least 2 million textual terms. The images are preprocessed according to scene to 512×512 pixel RGB format, resulting in a total dataset size of ≥7 million samples. This step primarily provides training data for power scenarios, covering multimodal inputs and scene diversity, supporting parallel model training.

[0023] S2. Split the dataset according to power scenarios and equipment types. In this embodiment, the dataset is divided into three subsets based on power plant equipment, distribution lines, and transformer area defect labels. Each GPU node is assigned an independent scenario subset to adapt the data to the model branches. This step splits the data according to power scenarios to avoid cross-scenario interference and achieve data parallelism that matches model requirements.

[0024] S3. Construct a hybrid parallel training framework. The specific construction method includes: (1) splitting the large power model into visual features and language features according to the visual branch and language branch to achieve model parallelism and reduce the computational load of a single card; (2) using fully sharded data parallelism to shard and store the model parameters, gradients and optimizer states on multiple cards to break through the bottleneck of single card memory; (3) using low-rank decomposition to compress the gradient of visual features and retaining the principal components; in addition, using non-uniform quantization to compress the gradient of language features and retaining the key term error of the gradient embedding of power terms not exceeding a preset error, thereby compressing the gradient data volume, reducing the cross-card communication time, and controlling the accuracy loss. In this way, the dual optimization of "memory usage + communication volume" can be achieved.

[0025] In this embodiment, the large multimodal power model is split into a visual branch, a language branch, and a fusion detection head, which are deployed on different GPU nodes to achieve model parallelism (MP). The visual branch is based on the Swin Transformer architecture, and the language branch is based on the BERT architecture. The parameters of the visual branch and the language branch account for 60% and 40% respectively, and the two branches are deployed on different nodes of multiple GPU clusters. The low-rank decomposition compression ratio is 4:1, retaining the first 25% of principal components, and the gradient quantization is 8-bit with a preset error of 2%. In addition, the visual features are compressed using singular value decomposition, and the language feature gradients are compressed using 8-bit non-uniform quantization, with dynamic switching adjusted according to the training stage.

[0026] In implementing the specific solution, the 1 billion parameter model can be split into a visual branch (Swin Transformer, 60% of parameters) and a language branch (BERT, 40% of parameters), deployed on different nodes of an 8-GPU A100 cluster. Furthermore, FSDP is used for even sharding across the 8 GPUs, with each GPU storing only 1 / 8 of the parameters (125 million parameters), reducing GPU memory usage to 25GB (at least 62% less than traditional DP). Additionally, visual gradients are decomposed using SVD low-rank decomposition (retaining the first 25% of principal components, compression ratio 4:1), and language gradients are quantized using 8-bit non-uniform quantization (key term error ≤2%). A dynamic on / off switch is adjusted according to the training stage (off in the early stage, on in the middle stage, and reduced compression ratio in the later stage). Specifically, compression is turned off in the early stage (first 100,000 steps) to ensure feature learning stability; compression is turned on in the middle stage (100,000-500,000 steps) to accelerate training; and the compression ratio is reduced to 2:1 in the later stage (after 500,000 steps) to ensure convergence accuracy.

[0027] S4. Prioritize the parallel execution of computationally intensive tasks (such as feature extraction) and communication-intensive tasks (such as gradient synchronization), hiding communication latency. The NCCL library is used to implement asynchronous gradient fragment transmission, reducing communication time from 40% to 25%. This step optimizes the cross-card communication strategy, hides communication latency, and improves parallel efficiency. In specific implementation, gradient fragments are asynchronously transmitted based on the NCCL library, employing a "computation-communication parallel" scheduling approach, reducing communication time from 40% to 25% and decreasing the communication volume of the 8-card cluster by 60%.

[0028] S5. Perform distributed training, utilizing the computing power of the GPU cluster to accelerate model convergence. In this step, an 8-card NVIDIA A100 cluster (312 TFLOPS per card) is deployed, interconnected via PCIe 4.0 NVLink, with training parameters of batch size 256, learning rate 2e-4, and 800,000 iterations.

[0029] S6. Monitor training speed and model accuracy metrics in real time and determine if they meet the standards. If yes, output the accelerated training model; otherwise, dynamically adjust the gradient compression rate and communication strategy according to the training phase and execute step S3. This step can monitor training effectiveness in real time, output key metrics, drive dynamic adjustments, dynamically optimize training strategies, and balance memory, speed, and accuracy.

[0030] In this embodiment, when the training speed (iterations per hour) increases by less than 20%, the gradient compression ratio is increased; when the model accuracy metric mAP decreases by more than 1%, the gradient compression ratio is decreased or the model branch parameter ratio is adjusted. Compression is disabled in the early stages of training, enabled in the middle stages, and the compression ratio is reduced to 2:1 in the later stages. In addition, when the speed is insufficient, communication scheduling is optimized (prioritizing the transmission of defect feature gradients), and when the GPU memory exceeds 30GB, the model branch parameter ratio is adjusted (visual:language = 55:45).

[0031] In summary, compared with existing large-scale power model training methods, the distributed training method for large-scale power models based on FSDP and gradient compression in this embodiment has the following advantages: 1. This distributed training method for large-scale power models based on FSDP and gradient compression introduces a hybrid parallel framework combining fully sharded data parallelism (FSDP) with power feature-aware gradient compression. It also incorporates scenario-based data sharding and dynamic feedback mechanisms to reduce single-card memory usage to below 30GB. Furthermore, it optimizes cross-card communication strategies, reduces communication volume through gradient compression, and designs a non-uniform compression algorithm tailored to power data characteristics to ensure that the model's mAP (memory accuracy) drops to a low level after compression. This solves the technical problems of excessive memory usage and slow training speed in existing large-scale power model training methods, achieving significant reductions in memory usage, improved training speed, and guaranteed model accuracy.

[0032] 2. This distributed training method for large-scale power models based on FSDP and gradient compression can reduce the single-card memory usage from 65GB to 25GB, supporting stable training of a 1 billion-parameter power model on an 8-card A100 cluster. Furthermore, the training cycle on the 8-card A100 cluster can be shortened from 14 days to 11.2 days, a 20% speed improvement. In addition, accuracy loss is controllable; gradient compression reduces the mAP by ≤1%, scenario-based data sharding improves defect detection recall by 3%, and non-uniform gradient compression protects terminology feature gradients, maintaining terminology recognition accuracy above 92%.

[0033] Example 2 This embodiment provides a distributed training system for a large power model based on FSDP and gradient compression. The system applies the distributed training method for a large power model based on FSDP and gradient compression in Embodiment 1, and includes a dataset construction module, a data sharding module, a framework construction module, a communication optimization module, a training module, and an evaluation feedback adjustment module.

[0034] The dataset construction module is used to build a multimodal power dataset, which is similar to or the same as the dataset in Example 1 and requires a sufficient data scale. The data sharding module is used to shard the dataset according to power scenarios and equipment types. Sharding is mainly based on label types, and can be divided into three subsets as in Example 1, or it can be adjusted according to actual needs.

[0035] The framework building module is used to construct a hybrid parallel training framework, and includes a model splitting unit, a parameter sharding unit, and a gradient compression unit. The model splitting unit splits the large power model into visual and linguistic features according to visual and linguistic branches. The parameter sharding unit stores model parameters, gradients, and optimizer states on multiple GPUs using a fully sharded data parallel approach. The gradient compression unit compresses the visual feature gradients using low-rank decomposition while retaining principal components. The gradient compression unit also performs non-uniform quantization compression on the linguistic feature gradients, and ensures that the key term error for power term embedding gradients does not exceed a preset error. Similarly, these modules or units can be implemented using the specific implementation methods described in Example 1.

[0036] The communication optimization module prioritizes the parallel execution of computationally intensive and communication-intensive tasks, employing the NCCL library for asynchronous gradient sharding. The training module performs distributed training, leveraging the computing power of GPU clusters to accelerate model convergence. The evaluation feedback adjustment module monitors training speed and model accuracy metrics in real time, determining whether they meet the standards. If so, it outputs an accelerated training model; otherwise, it dynamically adjusts the gradient compression rate and communication strategy based on the training phase and runs the framework building module. These modules can implement their respective functions using the specific solutions employed in the corresponding steps of Example 1. It should be noted that the system achieved in this embodiment achieves the same results as the method in Example 1, and will not be repeated here.

[0037] Example 3 This embodiment provides a computer terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the distributed training method for a large-scale power model based on FSDP and gradient compression described in Embodiment 1.

[0038] The method in Example 1 can be applied in software form, such as by designing it as a standalone program and installing it on a computer terminal, which can be a computer, smartphone, control system, or other IoT device. Alternatively, the method in Example 1 can be designed as an embedded program and installed on a computer terminal, such as on a microcontroller.

[0039] Example 4 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the steps of the distributed training method for a large power model based on FSDP and gradient compression as described in Embodiment 1.

[0040] When applying the method of Example 1, it can be applied in the form of software, such as by designing it as a program that can run independently on a computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, designed as a USB security token, and the program can be designed to start the entire method through an external trigger.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed training method for a large-scale power model based on FSDP and gradient compression, characterized in that, It includes the following steps: S1. Construct a multimodal power dataset; S2. Divide the dataset into segments according to power scenarios and equipment types; S3. Construct a hybrid parallel training framework, which includes: splitting the large power model into visual features and linguistic features according to visual and linguistic branches; storing model parameters, gradients, and optimizer states in multiple GPUs using a fully sharded data parallel approach; compressing the visual feature gradients using low-rank decomposition while retaining principal components; compressing the linguistic feature gradients using non-uniform quantization, and ensuring that the key term errors for the power term embedding gradients do not exceed a preset error. S4. Prioritize the parallel execution of computationally intensive and communication-intensive tasks, and use the NCCL library to implement gradient fragmentation and asynchronous transmission; S5. Perform distributed training and use the computing power of GPU clusters to accelerate model convergence; S6. Monitor the training speed and model accuracy in real time and determine whether they meet the standards. If they do, output the accelerated training model. Otherwise, dynamically adjust the gradient compression rate and communication strategy according to the training stage and execute step S3.

2. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, The vision branch is based on the Swing Transformer architecture, and the language branch is based on the BERT architecture.

3. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, In step S3, the compression ratio of the low-rank decomposition is 4:1, and the first 25% of the principal components are retained. The gradient quantization is 8 bits, and the preset error is 2%.

4. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, In step S6, compression is turned off in the early stage of training, turned on in the middle stage, and reduced to 2:1 in the later stage.

5. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, The dataset includes at least 5 million images of substation inspections and equipment close-ups, and at least 2 million text terms. The images are preprocessed by scene to 512×512 pixel RGB format.

6. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, In step S2, the system is divided into three subsets based on the defect labels of power station equipment, power distribution lines, and transformer substations. Each GPU node is assigned an independent scene subset to adapt the data to the model branches.

7. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, In step S3, the visual features are compressed using singular value decomposition, the language feature gradients are compressed using 8-bit non-uniform quantization, and the dynamic switch is adjusted according to the training stage.

8. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 1, characterized in that, In step S6, when the training speed improvement is less than 20%, the gradient compression ratio is increased; when the model accuracy index mAP decreases by more than 1%, the gradient compression ratio is decreased or the model branch parameter ratio is adjusted.

9. The distributed training method for a large-scale power model based on FSDP and gradient compression as described in claim 2, characterized in that, The parameters of the visual branch and the language branch account for 60% and 40% respectively, and the two branches are deployed on different nodes of multiple GPU clusters.

10. A distributed training system for a large-scale power model based on FSDP and gradient compression, characterized in that, Its application is the distributed training method for large-scale power models based on FSDP and gradient compression as described in any one of claims 1-9; the system includes: The dataset building module is used to build a multimodal power dataset. A data sharding module is used to shard the dataset according to power scenarios and equipment types; The framework construction module is used to build a hybrid parallel training framework, and includes a model splitting unit, a parameter sharding unit, and a gradient compression unit. The model splitting unit is used to split the large power model into visual features and linguistic features according to visual and linguistic branches. The parameter sharding unit is used to store model parameters, gradients, and optimizer states in shards on multiple GPUs using a fully sharded data parallel approach. The gradient compression unit is used to compress the gradients of the visual features using low-rank decomposition while retaining principal components. The gradient compression unit is also used to compress the gradients of the linguistic features using non-uniform quantization, and for the embedding gradients of power terms, it retains the key term error to be no more than a preset error. The communication optimization module is used to prioritize the parallel execution of computationally intensive and communication-intensive tasks, and uses the NCCL library to implement gradient fragmentation asynchronous transmission. The training module is used to perform distributed training, leveraging the computing power of GPU clusters to accelerate model convergence. The evaluation feedback adjustment module is used to monitor training speed and model accuracy metrics in real time and determine whether they meet the standards. If they do, it outputs a training acceleration model; otherwise, it dynamically adjusts the gradient compression rate and communication strategy according to the training phase and runs the framework construction module.

Images