Cross-data-center distributed training method, device and equipment and readable storage medium
Patent Information
- Application Number
- CN202611530513.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-30
- Publication Date
- 2026-11-03
AI Technical Summary
The computing power and power supply capacity of a single data center limit the training of ultra-large-scale models, and how to achieve distributed training across data centers has become an urgent problem to be solved.
By quantizing and compressing local training data, sending it to other data centers using a pre-created data transmission strategy, and storing and updating model gradients in a message queue, an optimal data transmission strategy is constructed to reduce network latency and jitter.
It achieves stable distributed training across data centers, reduces network latency and jitter, avoids the problem of asynchronous training efficiency in different data centers, and improves the stability and efficiency of training.
Smart Images
Figure CN123035239A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cloud computing technology, and in particular to a distributed training method, apparatus, device, and readable storage medium across data centers. Background Technology
[0002] In the context of intelligent computing, training larger-scale models typically requires tens of thousands or even hundreds of thousands of CPUs. However, single data centers currently face two main constraints: 1. A single data center is usually limited to a few hundred or a few thousand CPUs, with tens of thousands being relatively rare; 2. A single data center has limited power capacity, typically around 2MW, while large-scale intelligent computing usually requires 100MW. These two factors restrict the development of ultra-large-scale data centers, indicating that single data centers have limitations in computing resources and power capacity. To train larger-scale models, multiple data centers need to be utilized. Therefore, how to achieve distributed training across data centers has been a continuous research direction. Summary of the Invention
[0003] In view of this, this application provides a method, apparatus, device and readable storage medium for cross-data center distributed training, so as to facilitate cross-data center distributed training.
[0004] To achieve the above objectives, the following solution is proposed: A distributed training method across data centers includes: Complete the local training as instructed, and quantize and compress the local training data to obtain quantized and compressed training data. Based on a pre-created data transmission strategy, the quantized and compressed training data is sent to other data centers; It receives quantized and compressed training data sent from other data centers and stores it in a message queue; Determine whether the message queue has received all the quantized and compressed training data sent from other data centers; If so, the model gradient is updated using the local training data and the quantized compressed training data stored in the message queue; Determine if the model training is complete; If training is not completed, return to the step of completing local training according to the instructions and quantizing and compressing the local training data to obtain quantized and compressed training data.
[0005] Optionally, the local training data can be quantized and compressed to obtain quantized and compressed training data, including: The local training data is quantized and compressed according to a pre-set quantization compression standard to obtain preliminary quantized and compressed training data; The model performance loss was calculated using the preliminary quantized compressed data and local training data. Determine whether the performance loss of the model is within a pre-set error range; If not, the level is reduced based on the pre-set quantization compression standard, and the local training data is quantized and compressed according to the new quantization compression standard to obtain new preliminary quantization compressed training data. Then, the process returns to the step of using the preliminary quantization compressed data to calculate the model performance loss. If so, the preliminary quantized compressed training data will be used as the quantized compressed training data.
[0006] Optionally, the model performance loss can be calculated using the pre-quantized compressed data and local training data, including: The quantization performance loss was calculated using the preliminary quantization compression data. The basic performance loss is calculated using local training data; The model performance loss is calculated based on the quantization performance loss and the basic performance loss.
[0007] Optionally, the preset error range can be calculated using the following formula: Where N is the number of model parameters, D is the amount of data, and P is the model accuracy.
[0008] Optional, also includes: Determine the inspection interval; After the training step size reaches the test interval, the pre-set quantization compression standard is tested to obtain the test results; Based on the test results, adjust the pre-set quantization compression standard.
[0009] Optionally, the process of pre-creating a data transfer policy includes: Calculate the optimal path between data centers; For each optimal path, assign an identifier to each node on the path; Based on the identifier, an ordered list of segment identifiers indicating the data packet forwarding path and corresponding configurations are added to each node on the path to obtain a data transmission strategy, so that data transmission between data centers can be carried out according to the optimal path.
[0010] A distributed training device across data centers, comprising: The local training module is used to complete local training according to instructions and to quantize and compress the local training data to obtain quantized and compressed training data. The data transmission module is used to send quantized and compressed training data to other data centers based on a pre-created data transmission strategy; The data receiving module is used to receive quantized and compressed training data sent from other data centers and store it in a message queue; The data status judgment module is used to determine whether the message queue has received all the quantized and compressed training data sent by other data centers; if so, the model gradient is updated using the local training data and the quantized and compressed training data stored in the message queue. The training status judgment module is used to determine whether the model training is complete. If the training is not complete, it returns to execute the steps of completing local training according to the instructions and quantizing and compressing the local training data to obtain quantized and compressed training data.
[0011] Optionally, the local training module performs a process of quantizing and compressing the local training data to obtain quantized and compressed training data, including: The local training data is quantized and compressed according to a pre-set quantization compression standard to obtain preliminary quantized and compressed training data; The model performance loss was calculated using the preliminary quantized compressed data and local training data. Determine whether the performance loss of the model is within a pre-set error range; If not, the level is reduced based on the pre-set quantization compression standard, and the local training data is quantized and compressed according to the new quantization compression standard to obtain new preliminary quantization compressed training data. Then, the process returns to the step of using the preliminary quantization compressed data to calculate the model performance loss. If so, the preliminary quantized compressed training data will be used as the quantized compressed training data.
[0012] A distributed training device across data centers includes: a memory and a processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of a cross-data center distributed training method as described in any of the preceding claims.
[0013] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a cross-data center distributed training method as described in any of the preceding claims.
[0014] As can be seen from the above technical solutions, the cross-datacenter distributed training method provided in this application includes: completing local training according to instructions and quantizing and compressing the local training data to obtain quantized and compressed training data; sending the quantized and compressed training data to other data centers based on a pre-created data transmission strategy; receiving the quantized and compressed training data sent by other data centers and storing it in a message queue; determining whether the message queue has received all the quantized and compressed training data sent by other data centers; if so, updating the model gradient using the local training data and the quantized and compressed training data stored in the message queue; determining whether the model training is complete; if the training is not complete, returning to the step of completing local training according to instructions and quantizing and compressing the local training data to obtain quantized and compressed training data. This application achieves cross-datacenter distributed training by having each data center complete local training according to instructions and send the local training data to other data centers according to a pre-created data transmission strategy, while simultaneously receiving training data sent by other data centers.
[0015] Furthermore, by quantizing and compressing the local training data and transmitting it according to a pre-created data transmission strategy, network latency and jitter can be reduced to some extent, thus improving training stability. Simultaneously, building a message queue to store data sent from other data centers can avoid the problem of data asynchrony between different data centers due to varying training efficiencies. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0017] Figure 1 A flowchart of a cross-data center distributed training method provided in this application embodiment; Figure 2 A schematic diagram of a cross-data center distributed training device structure is provided in an embodiment of this application; Figure 3 This is a hardware structure block diagram of a cross-data center distributed training device provided in an embodiment of this application. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] Figure 1 A flowchart of a cross-datacenter distributed training method provided in this application embodiment may include the following steps: Step S100: Complete local training according to the instructions, and quantize and compress the local training data to obtain quantized and compressed training data.
[0020] Specifically, each data center needs to quantize and compress its local training data before sending it. The purpose of quantizing and compressing local training data is to reduce the amount of data transmitted, thereby reducing bandwidth pressure and speeding up transmission.
[0021] Step S101: Based on the pre-created data transmission strategy, send the quantized compressed training data to other data centers.
[0022] Specifically, before distributed training, each data center can first construct a data transmission path from source to destination, which serves as a pre-created data transmission strategy. This allows data transmission to proceed according to the pre-created data transmission strategy, following the planned forwarding path from source to destination. Compared to the step-by-step strategy in traditional networks, this can reduce latency and network jitter.
[0023] Step S102: Receive quantized compressed training data sent from other data centers and store it in a message queue.
[0024] Specifically, a message queue is created in each data center. After completing the corresponding local training according to the instructions, each data center broadcasts the generated local training data to other data centers. The message queue of each data center is used to receive training data sent from other data centers.
[0025] Creating a message queue in each data center can solve the problem of data asynchrony between different data centers. Because training paces vary between data centers (some faster, some slower), message queues can be built by allocating dynamic memory on the RDMA network card to act as message queues, achieving coupling between different data centers, smoothing data traffic, and enabling asynchronous processing. Message queues allow data centers to receive and aggregate data traffic, and finally asynchronously notify the upper-layer training service to perform calculations and update model gradient information.
[0026] Step S103: Determine whether the message queue has received all the quantized compressed training data sent by other data centers.
[0027] Specifically, if yes, then proceed to step S104; otherwise, continue receiving data.
[0028] Step S104: Update the model gradient using local training data and quantized compressed training data stored in the message queue.
[0029] Specifically, once the message queue receives all the training data sent by other data centers, it can asynchronously notify the upper-layer training service. The upper-layer application service then retrieves all the training data sent by other data centers from the message queue, combines it with the local training data, and updates the model gradient.
[0030] Step S105: Determine whether the model training is complete.
[0031] Specifically, if training is not completed, the process returns to the step of completing local training according to the instructions and quantizing and compressing the local training data to obtain quantized and compressed training data; if training is completed, step S106 is executed.
[0032] Step S106: Obtain the trained model.
[0033] As can be seen from the above technical solutions, the cross-datacenter distributed training method provided in this application includes: completing local training according to instructions and quantizing and compressing the local training data to obtain quantized and compressed training data; sending the quantized and compressed training data to other data centers based on a pre-created data transmission strategy; receiving the quantized and compressed training data sent by other data centers and storing it in a message queue; determining whether the message queue has received all the quantized and compressed training data sent by other data centers; if so, updating the model gradient using the local training data and the quantized and compressed training data stored in the message queue; determining whether the model training is complete; if the training is not complete, returning to the step of completing local training according to instructions and quantizing and compressing the local training data to obtain quantized and compressed training data. This application achieves cross-datacenter distributed training by having each data center complete local training according to instructions and send the local training data to other data centers according to a pre-created data transmission strategy, while simultaneously receiving training data sent by other data centers.
[0034] Furthermore, by quantizing and compressing the local training data and transmitting it according to a pre-created data transmission strategy, network latency and jitter can be reduced to some extent, thus improving training stability. Simultaneously, building a message queue to store data sent from other data centers can avoid the problem of data asynchrony between different data centers due to varying training efficiencies.
[0035] In some embodiments of this application, step S100, quantizing and compressing the local training data to obtain quantized and compressed training data, may include: S11. Quantize and compress the local training data according to the pre-set quantization compression standard to obtain preliminary quantized and compressed training data.
[0036] Specifically, each data center needs to quantize and compress its local training data before sending it. The purpose of quantization and compression is to reduce the amount of data transmitted, thereby reducing bandwidth pressure and speeding up transmission. The pre-set quantization and compression standard can be adjusted according to actual conditions.
[0037] S12. Using the preliminary quantized compressed data and local training data, calculate the model performance loss.
[0038] Specifically, assuming the pre-set quantization compression standard is 2 bits, quantization compression is performed according to the 2-bit quantization compression standard to obtain preliminary quantized compressed training data. Then, the unquantized compressed local training data is used to finally calculate the model performance loss.
[0039] S13. Determine whether the model performance loss is within the preset error range.
[0040] Specifically, if not, proceed to step S14; if yes, proceed to step S15. The pre-defined error range is calculated using the following formula: Where N is the number of model parameters, D is the amount of data, and P is the model accuracy.
[0041] S14. Based on the pre-set quantization compression standard, reduce the level and quantize and compress the local training data according to the new quantization compression standard to obtain new preliminary quantization compressed training data.
[0042] Specifically, assuming the pre-set quantization compression standard is 2-bit, quantization compression is performed according to the 2-bit standard to obtain preliminary quantized compressed training data. If the model performance loss is outside the pre-set error range when quantized according to 2-bit, the level of quantization compression is reduced from the pre-set standard, for example, from 2-bit to 4-bit, and the local training data is quantized and compressed according to the new 4-bit standard to obtain new preliminary quantized compressed training data. After obtaining the new preliminary quantized compressed training data, the process returns to the step of calculating the model performance loss using the preliminary quantized compressed data.
[0043] S15. Use the preliminary quantized compressed training data as the quantized compressed training data.
[0044] Quantization compression of training data aims to both accelerate transmission efficiency between data centers (e.g., 2-bit compression, 4-bit compression, and 8-bit compression can save 1 / 8, 1 / 4, and 1 / 2 of bandwidth resources respectively) and avoid progress loss due to over-compression. As model training time increases and the model converges, later stages may require quantization compression to 8-bit or even 16-bit. Although compression efficiency decreases, the impact of compression on model performance also decreases. Dynamically assessing the performance loss associated with compression allows for a trade-off between transmission efficiency and accuracy loss, achieving a better balance.
[0045] In some embodiments of this application, S12, calculating the model performance loss using preliminary quantized compressed data and local training data, may include: S21. Using the preliminary quantization compression data, calculate the quantization performance loss.
[0046] Specifically, assuming that preliminary quantization compression data is obtained by compressing according to the 2-bit quantization compression standard, the quantization performance loss is calculated using this preliminary quantization compression data.
[0047] S22. Calculate the basic performance loss using local training data.
[0048] Specifically, the basic performance loss is calculated using local training data that has not been quantized or compressed.
[0049] S23. Based on the quantization performance loss and the basic performance loss, the model performance loss is calculated.
[0050] Specifically, the model performance loss can be calculated by subtracting the basic performance loss from the quantified performance loss.
[0051] In some embodiments of this application, a 2-bit quantization compression standard can be used for quantization compression in the early stages of model training. As the process progresses, 4-bit compression may be used, followed by 8-bit compression. Finally, when the model training is nearing saturation, the original 16-bit bytes are used. This can reduce the performance degradation caused by quantization compression. Simultaneously, to ensure that the current quantization compression standard maximizes data compression while meeting the requirements for model performance loss, the current model quantization compression standard can be tested at certain intervals. Based on the test results, the quantization compression standard can be adjusted, for example, from 2-bit to 4-bit, from 4-bit to 8-bit, etc. Based on this, the above method may further include: S31. Determine the inspection interval.
[0052] Specifically, determining an appropriate testing interval can reduce testing frequency and unnecessary computation, while maximizing data compression using current quantization compression standards while meeting model performance loss requirements. Therefore, two strategies can be employed to determine the testing interval: the next testing period `step_next = 2*step_current` is determined with a probability of α = 0.8; the next sampling period `step_next = 0.4*step_current` is determined with a probability of 1 - α = 0.2, where `step_current` is the step size, initially set to 10000.
[0053] S32. After the training step size reaches the test interval, the pre-set quantization compression standard is tested to obtain the test results.
[0054] Specifically, after the training step size reaches the test interval, the pre-set quantization compression standard is tested. The test result can be that after compressing the training data using the current quantization compression standard, the model performance loss is within the pre-set error range, or after compressing the training data using the current quantization compression standard, the model performance loss is not within the pre-set error range.
[0055] S33. Adjust the pre-set quantitative compression standard based on the test results.
[0056] Specifically, if the model performance loss is within the preset error range after compressing the training data using the current quantization compression standard, then the preset quantization compression standard does not need to be adjusted; if the model performance loss is outside the preset error range after compressing the training data using the current quantization compression standard, then the quantization compression standard needs to be reduced, for example, from 2 bits to 4 bits, or from 4 bits to 8 bits, etc.
[0057] In some embodiments of this application, the process of pre-creating a data transmission strategy includes: S41. Calculate the optimal path between each data center.
[0058] Specifically, based on traditional interior gateway protocols such as OSPF and IS-IS, the optimal path between nodes is calculated. Through a distributed SDN controller, bandwidth, latency, and packet loss rate are calculated to determine the shortest path. The Dijkstra algorithm is then used to construct the shortest path for the entire full-mesh network.
[0059] S42. For each optimal path, assign an identifier to each node on the path.
[0060] Specifically, a SID identifier can be assigned to each node on the path.
[0061] S43. Based on the identifier, add an ordered list of segment identifiers indicating the packet forwarding path and the corresponding configuration to each node to obtain the data transmission strategy.
[0062] Specifically, assuming that the shortest path between data center 1 and data center 2 contains 4 nodes in addition to itself, assign SID identifiers to the 4 nodes, with nodes 1-4 being SID[1]-SID[4]. Simultaneously, an ordered segment list indicating the packet forwarding path is added to each node: Node 1 uses [2001:db8:1::5, 2001:db8:1::4, 2001:db8:1::3, 2001:db8:1::2], Node 2 uses [2001:db8:1::4, 2001:db8:1::3, 2001:db8:1::2], Node 3 uses [2001:db8:1::3, 2001:db8:1::2], and Node 4 uses [2001:db8:1::2]. This allows data transmission between data centers to follow a pre-determined optimal path, solving the problems of excessive latency and network jitter caused by the router's step-by-step approach in traditional networks. By reducing network bandwidth fluctuations caused by dynamic routing and lowering network latency, network jitter can be minimized, thereby improving training stability.
[0063] The following describes a cross-data center distributed training device provided in the embodiments of this application. The cross-data center distributed training device described below can be referred to in correspondence with the cross-data center distributed training method described above.
[0064] Figure 2 This application provides a schematic diagram of a cross-data center distributed training device structure, with reference to... Figure 2 As shown, the device may include: The local training module 10 is used to complete local training according to instructions and to quantize and compress the local training data to obtain quantized and compressed training data. The data transmission module 20 is used to send quantized compressed training data to other data centers based on a pre-created data transmission strategy; The data receiving module 30 is used to receive quantized compressed training data sent from other data centers and store it in a message queue. The data status judgment module 40 is used to determine whether the message queue has received all the quantized compressed training data sent by other data centers; if so, the model gradient is updated using the local training data and the quantized compressed training data stored in the message queue. The training status judgment module 50 is used to determine whether the model training is complete; if the training is not complete, it returns to the step of completing local training according to the instruction and quantizing and compressing the local training data to obtain quantized and compressed training data.
[0065] As can be seen from the above technical solution, the cross-data center distributed training device provided in this application includes: a local training module 10, used to complete local training according to instructions and quantize and compress the local training data to obtain quantized and compressed training data; a data sending module 20, used to send the quantized and compressed training data to other data centers based on a pre-created data transmission strategy; a data receiving module 30, used to receive the quantized and compressed training data sent by other data centers and store it in a message queue; a data status judgment module 40, used to judge whether the message queue has received all the quantized and compressed training data sent by other data centers; if so, the model gradient is updated using the local training data and the quantized and compressed training data stored in the message queue; and a training status judgment module 50, used to judge whether the model training is complete; if the training is not complete, the process returns to the step of completing local training according to instructions and quantizing and compressing the local training data to obtain quantized and compressed training data. This application realizes cross-data center distributed training by having each data center complete local training according to instructions, send the local training data to other data centers according to a pre-created data transmission strategy, and simultaneously receive training data sent by other data centers.
[0066] Furthermore, by quantizing and compressing the local training data and transmitting it according to a pre-created data transmission strategy, network latency and jitter can be reduced to some extent, thus improving training stability. Simultaneously, building a message queue to store data sent from other data centers can avoid the problem of data asynchrony between different data centers due to varying training efficiencies.
[0067] Optionally, the local training module may perform a process of quantizing and compressing the local training data to obtain quantized and compressed training data, which may include: The local training data is quantized and compressed according to a pre-set quantization compression standard to obtain preliminary quantized and compressed training data; The model performance loss was calculated using the preliminary quantized compressed data and local training data. Determine whether the performance loss of the model is within a pre-set error range; If not, the level is reduced based on the pre-set quantization compression standard, and the local training data is quantized and compressed according to the new quantization compression standard to obtain new preliminary quantization compressed training data. Then, the process returns to the step of using the preliminary quantization compressed data to calculate the model performance loss. If so, the preliminary quantized compressed training data will be used as the quantized compressed training data.
[0068] Optionally, the local training module performs a process of calculating the model performance loss using the pre-quantized compressed data and local training data, which may include: The quantization performance loss was calculated using the preliminary quantization compression data. The basic performance loss is calculated using local training data; The model performance loss is calculated based on the quantization performance loss and the basic performance loss.
[0069] Optionally, the distributed training apparatus across data centers may also include: Determine the inspection interval; After the training step size reaches the test interval, the pre-set quantization compression standard is tested to obtain the test results; Based on the test results, adjust the pre-set quantization compression standard.
[0070] Optionally, it may also include: a data transmission strategy creation module; The process of creating a data transmission policy by the data transmission policy creation module may include: Calculate the optimal path between data centers; For each optimal path, assign an identifier to each node on the path; Based on the identifier, an ordered list of segment identifiers indicating the data packet forwarding path and corresponding configurations are added to each node on the path to obtain a data transmission strategy, so that data transmission between data centers can be carried out according to the optimal path.
[0071] This application also provides a distributed training device across data centers. Figure 3 A hardware structure block diagram of a cross-datacenter distributed training device is shown, with reference to... Figure 3 The hardware structure of a cross-data center distributed training device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4; In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4; Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device; The memory stores a program, which the processor can call. The program is used to implement the various processing flows in the aforementioned cross-data center distributed training method.
[0072] This application embodiment also provides a storage medium that can store a program suitable for processor execution, the program being used to implement various processing flows in the aforementioned cross-data center distributed training method.
[0073] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0074] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined with each other, and the same or similar parts can be referred to each other.
[0075] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A distributed training method across data centers, characterized in that, include: Complete the local training as instructed, and quantize and compress the local training data to obtain quantized and compressed training data. Based on a pre-created data transmission strategy, the quantized and compressed training data is sent to other data centers; It receives quantized and compressed training data sent from other data centers and stores it in a message queue; Determine whether the message queue has received all the quantized and compressed training data sent from other data centers; If so, the model gradient is updated using the local training data and the quantized compressed training data stored in the message queue; Determine if the model training is complete; If training is not completed, return to the step of completing local training according to the instructions and quantizing and compressing the local training data to obtain quantized and compressed training data.
2. The method according to claim 1, characterized in that, The local training data is quantized and compressed to obtain quantized and compressed training data, including: The local training data is quantized and compressed according to a pre-set quantization compression standard to obtain preliminary quantized and compressed training data; The model performance loss was calculated using the preliminary quantized compressed data and local training data. Determine whether the performance loss of the model is within a pre-set error range; If not, the level is reduced based on the pre-set quantization compression standard, and the local training data is quantized and compressed according to the new quantization compression standard to obtain new preliminary quantization compressed training data. Then, the process returns to the step of using the preliminary quantization compressed data to calculate the model performance loss. If so, the preliminary quantized compressed training data will be used as the quantized compressed training data.
3. The method according to claim 2, characterized in that, Using the pre-quantized compressed data and local training data, the model performance loss was calculated, including: The quantization performance loss was calculated using the preliminary quantization compression data. The basic performance loss is calculated using local training data; The model performance loss is calculated based on the quantization performance loss and the basic performance loss.
4. The method according to claim 2, characterized in that, The preset error range is calculated using the following formula: Where N is the number of model parameters, D is the amount of data, and P is the model accuracy.
5. The method according to any one of claims 2-4, characterized in that, Also includes: Determine the inspection interval; After the training step size reaches the test interval, the pre-set quantization compression standard is tested to obtain the test results; Based on the test results, adjust the pre-set quantization compression standard.
6. The method according to claim 1, characterized in that, The process of pre-creating a data transfer policy includes: Calculate the optimal path between data centers; For each optimal path, assign an identifier to each node on the path; Based on the identifier, an ordered list of segment identifiers indicating the packet forwarding path and corresponding configurations are added to each node on the path to obtain a data transmission strategy, so that data transmission between data centers can be carried out according to the optimal path.
7. A distributed training device across data centers, characterized in that, include: The local training module is used to complete local training according to instructions and to quantize and compress the local training data to obtain quantized and compressed training data. The data transmission module is used to send quantized and compressed training data to other data centers based on a pre-created data transmission strategy; The data receiving module is used to receive quantized and compressed training data sent from other data centers and store it in a message queue; The data status judgment module is used to determine whether the message queue has received all the quantized compressed training data sent by other data centers; If so, the model gradient is updated using the local training data and the quantized compressed training data stored in the message queue; The training status determination module is used to determine whether the model has completed training; If training is not completed, return to the step of completing local training according to the instructions and quantizing and compressing the local training data to obtain quantized and compressed training data.
8. The apparatus according to claim 7, characterized in that, The local training module performs a process of quantizing and compressing local training data to obtain quantized and compressed training data, including: The local training data is quantized and compressed according to a pre-set quantization compression standard to obtain preliminary quantized and compressed training data; The model performance loss was calculated using the preliminary quantized compressed data and local training data. Determine whether the performance loss of the model is within a pre-set error range; If not, the level is reduced based on the pre-set quantization compression standard, and the local training data is quantized and compressed according to the new quantization compression standard to obtain new preliminary quantization compressed training data. Then, the process returns to the step of using the preliminary quantization compressed data to calculate the model performance loss. If so, the preliminary quantized compressed training data will be used as the quantized compressed training data.
9. A distributed training device across data centers, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of a cross-data center distributed training method as claimed in any one of claims 1-6.
10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a cross-data center distributed training method as claimed in any one of claims 1-6.