Model data loading method, device, equipment, storage medium and program product

Abstract

The disclosed embodiment discloses a model data loading method, apparatus, device, storage medium and program product, the model data loading method comprising: determining a second device participating in data loading from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed; the data loading capability is used to reflect the processing efficiency and the network bandwidth occupied by the first device when loading data; the model data to be processed is the relevant data of the model state saved at a specific time point during the training process; the model data to be processed is loaded to the second device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system through the second device. In this way, the efficiency of data loading and the bandwidth utilization can be significantly improved, and the robustness and reliability of the system can be enhanced.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, the field of computer technology, and in particular to a model data loading method, device, equipment, storage medium, and program product. Background Art

[0002] In machine learning, a checkpoint refers to data related to the state of a model saved at a specific point in time during the training process. In large-scale model training scenarios, all model data in the checkpoint needs to be loaded onto all devices involved in model training, and then model training is performed based on the loaded model data.

[0003] However, loading all model data onto each device will consume a large amount of computing resources, resulting in excessive consumption of network bandwidth and excessive waste of storage resources. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure at least provide a model data loading method, apparatus, device, storage medium and program product.

[0005] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0006] On the one hand, an embodiment of the present disclosure provides a model data loading method, which includes: determining a second device participating in data loading from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed; the data loading capability is used to reflect the processing efficiency and occupied network bandwidth of the first device when loading data; the model data to be processed is relevant data of the model state saved at a specific time point during the training process; the model data to be processed is loaded to the second device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system through the second device.

[0007] On the one hand, an embodiment of the present disclosure provides a model data loading method, which includes: after a second device participating in data loading in a distributed model training system obtains the model data to be processed, determining a fourth device participating in model training and a fifth device participating in data loading in the device group where the second device is located; the model data to be processed is relevant data of the model state saved at a specific time point during the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices, and the model data to be processed; the model data to be processed is divided based on the number of fourth devices and the number of fifth devices to obtain data blocks related to the training tasks of the fourth devices; each data block is distributed to the corresponding fourth device to achieve distribution of the model data to be processed to the devices participating in model training in the distributed model training system.

[0008] On the other hand, an embodiment of the present disclosure provides a model data loading device, which includes: a first determination module, used to determine a second device participating in data loading from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed; the data loading capability is used to reflect the processing efficiency and occupied network bandwidth of the first device when loading data; the model data to be processed is relevant data of the model state saved at a specific time point during the training process; and a loading module, used to load the model data to be processed to the second device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system through the second device.

[0009] On the other hand, an embodiment of the present disclosure provides a model data loading device, which includes: a second determination module, which is used to determine the fourth device participating in model training and the fifth device participating in data loading in the device group where the second device is located after the second device participating in data loading in the distributed model training system obtains the model data to be processed; the model data to be processed is the relevant data of the model state saved at a specific time point in the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed; a division module, which is used to divide the model data to be processed based on the number of fourth devices and the number of fifth devices, and obtain data blocks related to the training tasks of the fourth device; a distribution module, which is used to distribute each data block to the corresponding fourth device, so as to realize the distribution of the model data to be processed to the devices participating in model training in the distributed model training system.

[0010] On the other hand, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0011] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements part or all of the steps in the above method when executed by a processor.

[0012] On the other hand, an embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0013] On the other hand, an embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented.

[0014] In the disclosed embodiment, device selection is performed based on the data loading capability of the first device included in the distributed model training system and the model data to be processed, and the second device with the best device performance is selected from the first device, and data loading and data distribution are performed through the second device; thus, by loading the complete model data on a limited number of devices, the efficiency of data loading and bandwidth utilization can be significantly improved; accurately distributing the data blocks related to the training tasks of each device to each device can reduce unnecessary data transmission and further improve the utilization of network bandwidth. Thus, not only can the efficiency of data loading and bandwidth utilization be significantly improved, but also the robustness and reliability of the system are enhanced, making large-scale machine learning training more efficient and economical.

[0015] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0017] Figure 1 A schematic diagram of an implementation process of a model data loading method provided in an embodiment of the present disclosure Figure 1 ;

[0018] Figure 2 A schematic diagram of an implementation process of a model data loading method provided in an embodiment of the present disclosure Figure 2 ;

[0019] Figure 3 A schematic diagram of an implementation process of a model data loading method provided in an embodiment of the present disclosure Figure 3 ;

[0020] Figure 4 A schematic diagram of an implementation process of a model data loading method provided in an embodiment of the present disclosure Figure 4 ;

[0021] Figure 5 A schematic diagram of an implementation process of a model data loading method provided in an embodiment of the present disclosure Figure 5 ;

[0022] Figure 6 A schematic diagram of the structure of a model data loading device provided in an embodiment of the present disclosure Figure 1 ;

[0023] Figure 7 A schematic diagram of the structure of a model data loading device provided in an embodiment of the present disclosure Figure 2 ;

[0024] Figure 8 A hardware entity schematic diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0026] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0029] In order to better understand the model data loading method provided by the embodiment of the present disclosure, the solution in the related art is first described below.

[0030] In the large-scale pre-training process in the field of machine learning, the loading of checkpoints usually requires loading all the model data and optimizer states onto each device (such as GPU or CPU). This approach requires copying the entire model data on each device, and then loading the required data segments based on the specific role and needs of each device, and discarding other irrelevant data. This process consumes a lot of computing resources, and in a multi-device environment, especially in a cloud infrastructure, there is a significant waste of network bandwidth and storage resources.

[0031] For example, traditional large-scale training systems may need to synchronize hundreds to thousands of GPUs, during which each GPU will download complete model parameters from a central storage system, regardless of whether these parameters are ultimately used by the GPU. Although this approach is feasible in small-scale systems, in large-scale training systems, as the number of devices increases, its inefficiency and high resource consumption become obvious.

[0032] There are several problems with the related technology:

[0033] 1. Bandwidth waste: In a multi-device environment, especially when using cloud computing resources, unnecessary data copying and transmission consumes a lot of network bandwidth and increases the marginal cost of training.

[0034] 2. Low memory utilization: Since each device is loaded with full data, much of the data is actually not used, resulting in extremely low memory resource utilization.

[0035] 3. Startup delay: Before the actual training begins, a large amount of data must be loaded and synchronized, which is particularly prominent when there are many devices, thus prolonging the training startup time.

[0036] To this end, the embodiments of the present disclosure provide a model data loading method, which reduces unnecessary data transmission and improves the efficiency of memory and bandwidth utilization by optimizing the data loading and distribution mechanism, thereby shortening the startup time, reducing costs, and improving the overall efficiency of large-scale training.

[0037] The model data loading method provided in the embodiment of the present disclosure can be applied to the main control device of the distributed model training system. Figure 1 As shown, the method includes the following steps 101 to 102:

[0038] Step 101: Based on the data loading capability of the first devices included in the distributed model training system, the topological structure between the first devices and the model data to be processed, determine the second device participating in data loading from the first devices.

[0039] Among them, the data loading capability is used to reflect the processing efficiency and occupied network bandwidth of the first device when loading data; the model data to be processed is the relevant data of the model state saved at a specific time point in the training process.

[0040] The distributed model training system is used to process model training tasks using distributed deployed devices to significantly improve the speed and efficiency of training large and complex models. The distributed model training system may include multiple training devices and multiple storage devices. The training devices are used to perform model training, and the storage devices are used to store the model data generated during the model training process.

[0041] The first device refers to all devices included in the distributed model training system. The second device refers to a part of the devices selected from the first device for participating in the initial data loading. The data loading capacity is used to reflect the processing efficiency and network bandwidth occupied by each device when loading data. The topological structure between the first devices is used to reflect the network connection relationship between the devices in the distributed model training system.

[0042] The model data to be processed is the data related to the model state saved at a specific time point during the training process, that is, the model data to be processed refers to the data related to the model state saved by the checkpoint. For example, the model data to be processed may include but is not limited to: model parameters such as weights and biases, optimizer states such as learning rate and momentum, training rounds, performance indicators such as loss value and accuracy, etc.

[0043] Generally, the relevant data of the model state saved by the checkpoint is stored in a distributed storage system. Therefore, when the checkpoint is loaded in a distributed manner, a set of input / output (I / O) devices is shared. At this time, if all devices participate in the checkpoint loading, the overall I / O speed will be reduced; if a single device loads the checkpoint, it may not be able to fully occupy the overall I / O bandwidth of the system, resulting in bandwidth waste. To solve this problem, the embodiment of the present disclosure selects a device based on the data loading capability of the first device included in the distributed model training system and the model data to be processed, selects a second device with the best device performance from the first device, and loads data through the second device; in this way, the efficiency of data loading and bandwidth utilization can be significantly improved.

[0044] In some implementations, the first devices may be sorted according to the topological structure between the first devices, the data loading capabilities of the first devices, and the model data to be processed, and the first k first devices in the sorting order may be used as the second devices.

[0045] In some implementations, the second device that participates in data loading can be selected from the first device based on the topological structure between the first devices, the data loading capability of the first device, the hardware facilities of the first device, and the model data to be processed. The hardware facilities may include but are not limited to: a processor, memory, storage, interface, and power module.

[0046] Step 102: Load the model data to be processed into the second device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system through the second device.

[0047] The model data to be processed is first loaded into the second device, and then the model data to be processed is distributed to the devices participating in the model training in the distributed model training system through the second device; in this way, not only the efficiency of data loading and bandwidth utilization can be significantly improved, but also the robustness and reliability of the system can be enhanced, making large-scale machine learning training more efficient and economical.

[0048] In some implementations, the second device may obtain the model data to be processed from the distributed storage system to perform complete loading of the model data.

[0049] In some embodiments, after receiving the model data to be processed, the second device can divide the model data to be processed according to the data requirements of the training tasks of each device participating in the model training, obtain data blocks related to the training tasks of each device, and distribute the data blocks related to the training tasks of each device to each device.

[0050] In some embodiments, after receiving the model data to be processed, the second device can divide the model data to be processed according to the data requirements of the training tasks of each device participating in the model training, the devices participating in data loading corresponding to the second device, and the devices participating in model training corresponding to the second device, to obtain data blocks related to the training tasks of each device, and distribute the data blocks related to the training tasks of each device to each device.

[0051] It should be noted that accurately distributing the data blocks related to the training tasks of each device to each device can reduce unnecessary data transmission and improve the utilization of network bandwidth.

[0052] In the disclosed embodiment, device selection is performed based on the data loading capability of the first device included in the distributed model training system and the model data to be processed, and the second device with the best device performance is selected from the first device, and data loading and data distribution are performed through the second device; thus, by loading the complete model data on a limited number of devices, the efficiency of data loading and bandwidth utilization can be significantly improved; accurately distributing the data blocks related to the training tasks of each device to each device can reduce unnecessary data transmission and further improve the utilization of network bandwidth. Thus, not only can the efficiency of data loading and bandwidth utilization be significantly improved, but also the robustness and reliability of the system are enhanced, making large-scale machine learning training more efficient and economical.

[0053] The present disclosure provides a model data loading method, which can be applied to a second device (loading device) of a distributed model training system. Figure 2 As shown, the method includes the following steps 201 to 203:

[0054] Step 201: After the second device participating in data loading in the distributed model training system obtains the model data to be processed, a fourth device participating in model training and a fifth device participating in data loading in the device group where the second device is located are determined.

[0055] Among them, the model data to be processed is the relevant data of the model state saved at a specific time point in the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed.

[0056] The second device refers to a device that participates in data loading. The fourth device refers to all devices that participate in model training in the device group where the second device is located. The fifth device refers to a device that participates in data loading in the device group where the second device is located.

[0057] Step 202: divide the model data to be processed based on the number of the fourth devices and the number of the fifth devices to obtain data blocks related to the training task of the fourth device.

[0058] Dividing the model data to be processed based on the number of fourth devices and the number of fifth devices means that data is segmented according to the number of devices participating in model training and the number of devices participating in data loading in each device group, so as to maximize the use of the network bandwidth of each device, reduce redundant transmission of data between devices, and improve data transmission efficiency and the reliability of data segmentation.

[0059] In some implementations, the number of divisions of the model data to be processed can be determined based on the number of fourth devices and the number of fifth devices; the model data to be processed is divided according to the number of divisions to obtain data blocks related to the training task of the fourth device.

[0060] In some implementations, the number of divisions of the model data to be processed can be determined based on the number of fourth devices and the number of fifth devices; the division of the model data to be processed is guided according to the data requirements of the training task of the fourth device to obtain the number of divisions, that is, data blocks.

[0061] It should be noted that the data block related to the training task of the fourth device is only a small part of the model data to be processed, which greatly optimizes the data flow of the system network and reduces bandwidth requirements.

[0062] Step 203: distribute each of the data blocks to the corresponding fourth device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system.

[0063] In some embodiments, a device-to-device (D2D) communication method can be used to distribute each data block from the second device (loading device) to the corresponding fourth device (training device) so that the fourth device performs model training based on the received data block.

[0064] In the disclosed embodiment, data segmentation is performed according to the number of devices participating in model training (fourth devices) and devices participating in data loading (fifth devices) in each device group, so that the network bandwidth of each device can be utilized to the maximum extent, redundant data transmission between devices can be reduced, and data transmission efficiency and reliability of data segmentation can be improved; and during transmission, only data blocks related to the training task of the fourth device are transmitted, rather than all model data, which can greatly optimize the data flow of the system network and reduce bandwidth requirements. In this way, the robustness and reliability of the system are further enhanced, making large-scale machine learning training more efficient and economical.

[0065] The present disclosure provides a model data loading method, which can be applied to a main control device of a distributed model training system. Figure 3 As shown, the method includes the following steps 301 to 303:

[0066] Step 301: Group the first devices based on the topological structure between the first devices to obtain multiple device groups.

[0067] The purpose of grouping the first devices based on the topological structure between the first devices is to divide the devices with topological connection into one group to improve the reliability of data distribution.

[0068] In some implementations, step 301 may be specifically implemented by: determining a topological connection between a training device and a storage device in the first device; and dividing the training devices and storage devices having the topological connection into a group to obtain a plurality of device groups.

[0069] In a feasible implementation, a topology recognition algorithm can be used to identify the topological connections between training devices and storage devices in a distributed model training system, and the network-interconnected storage devices and training devices are grouped together to obtain multiple device groups.

[0070] Step 302: Determine the second device from a plurality of the device groups based on the model data to be processed and the data loading capability of each device in the device group.

[0071] Among them, the data loading capability is used to reflect the processing efficiency and occupied network bandwidth of the first device when loading data; the model data to be processed is the relevant data of the model state saved at a specific time point in the training process.

[0072] Here, the above steps 301 to 302 correspond to the above step 101, and the specific implementation of the above step 101 may be referred to during implementation.

[0073] In some embodiments, the specific implementation method of step 302 may be: based on the number of files storing model data to be processed and the number of device groups, determine a first number of devices in the device group that participate in data loading; based on the first number and the data loading capability of each device in the device group, determine a second device from the module group.

[0074] Since the model data in large-scale model training scenarios is relatively large and needs to be stored in multiple files, when determining the loading device, the number of files of the model data to be processed and the number of device groups can be considered to determine the first number of second devices that maximizes both network bandwidth utilization and data loading rate.

[0075] Further, the specific implementation method of step 302 may be: determining the number of files storing the model data to be processed; based on the number of the files and the number of the device groups, determining the first number of devices in the device group that participate in data loading; sorting the training devices in the device group based on the data loading capability of each device in the device group; and using the first number of training devices in the device group that are sorted first as the second devices.

[0076] Using the first number of training devices in the device group as the second devices can further maximize both network bandwidth utilization and data loading rate.

[0077] In a feasible implementation, the specific implementation method of determining the data loading capability of each device in the device group may be: loading the test data to each device group at the same time, and testing the data loading capability of each device in each device group.

[0078] In a feasible implementation, the number of device groups can be represented as n, and the number of files can be represented as m. At this time, the number of devices participating in data loading in each device group (first number) can be selected in the range of 0 to m / n, and the training device with the best data loading capability in each device group is used as the second device.

[0079] Step 303: load the model data to be processed into the second device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system through the second device.

[0080] Here, the above step 303 corresponds to the above step 102, and the specific implementation of the above step 102 may be referred to during implementation.

[0081] Based on the above embodiments, the model data loading method provided in the embodiments of the present disclosure can also optimize the devices in the device group that participate in data loading through the following steps 304 to 305:

[0082] Step 304: After each round of data loading is completed, determine the performance data of the distributed model training system.

[0083] The performance data of the distributed model training system is used to evaluate the operation of the distributed model training system in each round. The performance data may include but is not limited to: CPU usage, memory usage, frequency of disk I / O operations, network bandwidth, response time, and system load.

[0084] Since the system cannot remain stable all the time in large-scale model training scenarios, after each round of data loading, the system performance data can be collected to evaluate the implementation effect of the current solution through the system performance data.

[0085] In a feasible implementation, a performance analysis tool may be used to collect performance data of a distributed model training system.

[0086] Step 305: Based on the performance data, adjust the devices in the device group that participate in data loading.

[0087] The purpose of adjusting the devices involved in data loading in the device group based on performance data is to continuously optimize the data loading strategy and data distribution strategy according to the implementation effect of the current plan, so as to adapt to training needs of different scales and complexities and continuously improve the overall efficiency and response speed of the system.

[0088] In a feasible implementation, the number (first number) of devices in the device group that participate in data loading may be adjusted based on the performance data.

[0089] In some embodiments, the specific implementation method of step 305 may be: determining whether the network bandwidth in the performance data in the target round is lower than the bandwidth threshold; when the network bandwidth in the target round is lower than the bandwidth threshold, adjusting the first number based on the number of devices participating in data loading in the target round and the network bandwidth.

[0090] The target round can be preset and can be set according to the business scenario; for example, the target round can be set to 3. The bandwidth threshold is used to determine whether the network bandwidth meets the expected effect. The bandwidth threshold can also be set according to the business scenario; for example, the bandwidth threshold can be set to full bandwidth.

[0091] In a feasible implementation, the network bandwidth may refer to the I / O bandwidth. If the I / O bandwidth is lower than the full bandwidth in three rounds, the number of devices in the device group participating in data loading is adjusted.

[0092] In some embodiments, the specific implementation method of "adjusting the first number based on the number of devices participating in data loading in the target round and the network bandwidth" can be: performing digital fitting on the number of devices participating in data loading in the target round and the network bandwidth to determine the optimal number of devices participating in data loading; and adjusting the first number to the optimal number.

[0093] In a feasible implementation method, a quadratic function can be fitted between the number of devices participating in data loading in three rounds and the I / O bandwidth, and the optimal number can be selected to participate in the next round of checkpoint loading. I / O performance data can be continuously collected and iterated.

[0094] Based on the above embodiments, the model data loading method provided in the embodiments of the present disclosure can also perform fault detection on the model data loading process and distribution process through the following steps 306 to 307:

[0095] Step 306: Perform fault detection on the loading process of the model data to be processed.

[0096] In a feasible implementation, a fault detection mechanism may be used to perform fault detection on the loading process of the model data to be processed.

[0097] Specifically, the integrity and / or consistency of the model data to be processed can be detected based on the data to be detected returned by the device participating in data loading. The data to be detected can be part or all of the model parameters, or the optimizer state, etc.

[0098] In a feasible implementation, PyTorch's gather communication operator can be used to collect the data to be tested from all devices involved in data loading, and compare the data to be tested with the model data to be processed to see if they are consistent. If they are consistent, it means that there is no abnormality in the data loading process; if they are inconsistent, it means that there is an abnormality in the data loading process.

[0099] In a feasible implementation, a flag can be used as an identifier, with an initial value of 0, which is set to 1 after data distribution is completed; the All-reduce algorithm is used to merge the flags of all devices participating in data loading. If the merged flags are all 1, and the number of flags is equal to the number of devices participating in data loading, it means that data loading is performed under data consistency; if the merged flag is not 1, and / or the number of flags is not equal to the number of devices participating in data loading, it means that data loading is not performed under data consistency.

[0100] Step 307: Perform fault detection on the distribution process of the model data to be processed.

[0101] In a feasible implementation, a fault detection mechanism may be used to perform fault detection on the distribution process of the model data to be processed.

[0102] Specifically, the integrity and / or consistency of the model data to be processed can be detected based on the data to be detected sent back by the device participating in the model training. The data to be detected can be part or all of the model parameters, or the optimizer state, etc.

[0103] In some embodiments, the specific implementation method of step 307 may be: based on the first data to be detected sent back by the third device participating in the model training, the integrity of the model data to be processed is detected; based on the second data to be detected sent back by the third device, the synchronization of multiple third devices is detected; when both the integrity and the synchronization are detected successfully, it is determined that there is no abnormality in the distribution process of the model data to be processed; when at least one of the integrity and the synchronization fails to be detected, it is determined that there is an abnormality in the distribution process of the model data to be processed.

[0104] The first detection data refers to part or all of the model data received by the third device participating in the model training. The second data to be detected may refer to a flag.

[0105] In some embodiments, the specific implementation method of "detecting the integrity of the model data to be processed based on the first data to be detected sent back by the third device participating in the model training" can be: receiving the model data to be verified sent back by the third device; the first data to be detected includes the model data to be verified; determining whether the model data to be verified matches the model data to be processed; if any of the model data to be verified does not match the model data to be processed, determining that the integrity check has failed; if both the model data to be verified and the model data to be processed match, determining that the integrity check is successful.

[0106] In a feasible implementation, PyTorch's gather communication operator can be used to collect the model data to be verified sent back by all devices participating in model training, and compare the model data to be verified with the model data to be processed to see if they are consistent. If they are consistent, it means that there is no abnormality in the data distribution process; if they are inconsistent, it means that there is an abnormality in the data distribution process.

[0107] In some embodiments, the specific implementation method of "detecting the synchronization of multiple third devices based on the second data to be detected sent back by the third device" can be: obtaining an identifier corresponding to the third device; the second data to be detected includes the identifier; the identifier is used to indicate whether the data block has been received by the corresponding third device; when the number of the identifiers is equal to the number of the third devices and the identifiers are all first values, determining that the synchronization detection is successful; the first value indicates that the data block has been received by the corresponding third device; when the number of the identifiers is not equal to the number of the third devices, and / or any of the identifiers is a second value, determining that the distribution of the model data to be processed has failed; the second value indicates that the data block has not been received by the corresponding third device.

[0108] The first value may be 1, and the second value may be 0.

[0109] In a feasible implementation, a flag can be used as an identifier, with an initial value of 0, which is set to 1 after data reception is completed; the All-reduce algorithm is used to merge the flags of all devices participating in model training. If the merged flags are all 1 and the number of flags is equal to the number of devices participating in model training, it means that data distribution is carried out under data consistency; if the merged flag is not 1 and / or the number of flags is not equal to the number of devices participating in model training, it means that data distribution is not carried out under data consistency.

[0110] In a feasible implementation, during the data distribution process (or data loading process), if it is found during integrity verification and checking that a certain segment of the model and optimizer state is inconsistent with the original data (model data to be processed) or damaged, the system will automatically reload the required data segment from the data source or cache, and repeat the data distribution (data loading) and verification steps to ensure the robustness of the system; if the number of flags is not the number of devices for a long time during the synchronization and consistency check process, the I / O operation of the corresponding device will be re-executed to ensure the continuity of training.

[0111] In the disclosed embodiment, by loading complete model data on a limited number of devices, the efficiency of data loading and bandwidth utilization can be significantly improved; accurately distributing data blocks related to the training task of each device to each device can reduce unnecessary data transmission and further improve the utilization of network bandwidth. Thus, not only can the efficiency of data loading and bandwidth utilization be significantly improved, but also the robustness and reliability of the system are enhanced, making large-scale machine learning training more efficient and economical. By accurately controlling which data is loaded and how it is distributed, the data preparation time can be effectively shortened and the overall efficiency of data processing can be improved. This efficiency improvement is achieved by reducing redundant transmission of data between devices and optimizing the data loading process. Through the data synchronization and consistency verification mechanism, it is ensured that all devices have consistent and complete data before the start of training. This mechanism greatly reduces training errors caused by inconsistent data and improves the reliability of model training. By introducing a fault detection and automatic recovery mechanism, problems can be automatically identified and repaired during data loading or distribution, ensuring the continuity of training and data integrity, enhancing the robustness of the system, and reducing the need for manual intervention. The model data loading method provided by the embodiment of the present disclosure is highly scalable and can flexibly adapt to training requirements from small-scale to large-scale industrial level, which makes the method efficient and economical in a variety of different environments.

[0112] The present disclosure provides a model data loading method, which can be applied to a second device (loading device) of a distributed model training system. Figure 4As shown, the method includes the following steps 401 to 404:

[0113] Step 401: After the second device participating in data loading in the distributed model training system obtains the model data to be processed, a fourth device participating in model training and a fifth device participating in data loading in the device group where the second device is located are determined.

[0114] Among them, the model data to be processed is the relevant data of the model state saved at a specific time point in the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed.

[0115] Here, the above step 401 corresponds to the above step 201, and the specific implementation of the above step 201 may be referred to during implementation.

[0116] In some implementations, after obtaining the model data to be processed, anomaly detection may be performed on the model data to be processed to determine whether there is an anomaly in the transmission process of the model data to be processed.

[0117] Step 402: Determine the number of divisions of the to-be-processed model data based on the number of the fourth devices and the number of the fifth devices.

[0118] The fourth device refers to the device in the device group that participates in model training, and the fifth device refers to the device in the device group that participates in data loading.

[0119] In a feasible implementation, a division operation may be performed on the number of fourth devices and the number of fifth devices to obtain the number of divisions of the model data to be processed.

[0120] For example, the number of the fourth device can be expressed as d, and the number of the fifth device can be expressed as t. In this case, the calculation formula for the divided number can be expressed as: d / t.

[0121] Step 403: divide the model data to be processed based on the number of divisions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

[0122] In a feasible implementation, the model data to be processed may be divided according to a multiple relationship between the total data volume of the model data to be processed and the number of divisions, to obtain data blocks related to the training task of each fourth device.

[0123] In some implementations, a specific implementation of step 403 may be: determining whether the total data volume of the model data to be processed is divisible by the number of divisions; if the total data volume is divisible by the number of divisions, dividing the model data to be processed based on the number of divisions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device;

[0124] Then, when the total data volume cannot be divided by the number of divisions, the model data to be processed is padded with preset characters to obtain the padded model data to be processed; the data volume of the padded model data to be processed is divisible by the number of divisions, and the preset characters will not affect the specific content of the model data to be processed; the padded model data to be processed is divided based on the number of divisions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

[0125] If the total amount of data cannot be divided by d / t, padding is used. The padding method used in the embodiment of the present disclosure is zero padding, that is, the model data to be processed is padded with 0 to make it divisible by d / t, and the data range to be received by each device is recorded so that the padded part can be discarded after the data reception is completed according to the data range.

[0126] Step 404: distribute each of the data blocks to the corresponding fourth device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system.

[0127] In the disclosed embodiment, data segmentation is performed according to the number of devices participating in model training (fourth devices) and devices participating in data loading (fifth devices) in each device group, so that the network bandwidth of each device can be utilized to the maximum extent, redundant data transmission between devices can be reduced, and data transmission efficiency and reliability of data segmentation can be improved; and during transmission, only data blocks related to the training task of the fourth device are transmitted, rather than all model data, which can greatly optimize the data flow of the system network and reduce bandwidth requirements. In this way, the robustness and reliability of the system are further enhanced, making large-scale machine learning training more efficient and economical.

[0128] The present disclosure provides a model data loading method, which can be applied to a distributed model training system. Figure 5 As shown, the method includes the following steps 501 to 507:

[0129] Step 501: The main control device groups the first devices based on the topological structure between the first devices included in the distributed model training system to obtain multiple device groups.

[0130] Step 502: The main control device determines a second device from a plurality of device groups based on the model data to be processed and the data loading capability of each device in the device group.

[0131] Step 503: The main control device loads the model data to be processed into the second device.

[0132] Step 504: The second device determines a fourth device participating in model training and a fifth device participating in data loading in the device group to which the second device belongs.

[0133] Step 505: The second device determines the number of divisions of the to-be-processed model data based on the number of the fourth devices and the number of the fifth devices.

[0134] Step 506: The second device divides the model data to be processed based on the number of divisions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

[0135] Step 507: The second device distributes each data block to the corresponding fourth device, thereby distributing the model data to be processed to all devices participating in model training in the distributed model training system.

[0136] The following describes the application of the model data loading method provided by the embodiment of the present disclosure in actual scenarios.

[0137] The disclosed embodiment proposes a new checkpoint loading method (corresponding to the above-mentioned model data loading method) to improve the data processing efficiency of large-scale pre-trained models. The specific scheme of the method is as follows:

[0138] 1. Dynamic device selection;

[0139] The number of devices involved in the initial data loading is dynamically determined through a specially designed evaluation model. The evaluation model determines the number of devices involved in the initial data loading based on the number of model file shards and the current storage system performance (such as transmission speed and transmission preemption method, etc.). The number of devices should be selected from the divisors of the number of model file shards to ensure the symmetry of the devices and avoid wasting the bandwidth of the storage system. The evaluation model takes into account the current network bandwidth, the computing power of the device, and the overall scale of the pre-training task to maximize resource allocation and reduce unnecessary data transmission.

[0140] 2. Load segmented data;

[0141] The entire model and optimizer data are loaded on limited devices in advance. These limited devices serve as the central nodes for data distribution, not only taking on the responsibility of data storage, but also handling the initial verification and preprocessing of data to prepare for data distribution.

[0142] 3. Efficient data distribution;

[0143] The scatter communication operator in the PyTorch framework is used to implement device-to-device (D2D) parameter data distribution. Each device only receives the part of the model it needs, greatly reducing the overall data transmission requirements and loading time.

[0144] 4. Data synchronization after distribution;

[0145] After the distribution is completed, the consistency of model parameters on all devices is ensured through a centralized or decentralized data synchronization mechanism. This step is crucial to maintaining the stability and effectiveness of model training.

[0146] 5. Data integrity verification;

[0147] The gather communication operator collects the model parameters of all devices back to the master device for verification. This step ensures that the data is not damaged or loaded incorrectly during transmission, increasing the reliability of the system.

[0148] 6. Fault recovery mechanism;

[0149] A fault recovery mechanism is designed to deal with any failures that may occur during data loading or distribution. If data errors or loss are found, the system can re-acquire the missing data from the central node to ensure training continuity and data integrity.

[0150] 7. Scalability and adaptability;

[0151] The disclosed embodiments take into account pre-training requirements of different scales and can flexibly adapt to different training environments ranging from dozens of devices to thousands of devices. By dynamically adjusting the selection parameters of these limited devices, small-scale or extremely large-scale data loading requirements can be optimized.

[0152] The implementation of the disclosed embodiments not only significantly improves the efficiency of data loading and bandwidth utilization, but also enhances the robustness and reliability of the system, making large-scale machine learning training more efficient and economical.

[0153] For a large-scale distributed training environment, the specific implementation process and details of the embodiment of the present disclosure are as follows:

[0154] 1. Deployment evaluation model;

[0155] Before the pre-training task starts, a machine learning model is deployed to evaluate and decide which devices should participate in the initial data loading. The model evaluation is calculated based on the available network bandwidth, the processing power of each device, and the total size of the task. In the current large-scale distributed training scenario, the model is usually stored in a distributed storage system, and the distributed loading checkpoints share the same set of I / O devices. Therefore, if all devices participate in checkpoint loading, it will bring challenges to the congestion algorithm and reduce the overall I / O speed. If a single device loads a checkpoint, it may not be able to fully occupy the overall I / O bandwidth of the system, resulting in bandwidth waste.

[0156] This evaluation model mainly includes the following parts: 1. Topology identification algorithm; This topology identification algorithm is used to identify the topological connection between the training device and the distributed storage system. Usually, several training devices are connected to a part of the storage system, and the systems are connected to each other through the network to form a distributed storage system. 2. I / O device determination algorithm; This I / O device determination algorithm divides the training devices corresponding to the various parts of the storage system connected to each other through the network into a group according to the identified topological structure, and the number of groups is n, and the number of model shards to be loaded is m. At this time, the number of distributed loads executed in each group is selected in the range of 0 to m / n. The selection method is that the devices in each group load data of a fixed size at the same time, and the number of devices that perform loading at the same time is determined to achieve the optimal loading efficiency. The evaluation model runs on one or more dedicated management nodes, which are responsible for monitoring the resource usage of the entire network and adjusting the computing strategy in real time.

[0157] 2. Initial data loading and preprocessing;

[0158] According to the number of loading devices in each group output by the evaluation model, t devices with the highest ranking are selected, and these devices are selected for complete data loading of the model and optimizer. These devices will obtain the data of the model and optimizer from the distributed storage system and perform necessary preprocessing steps, including initial analysis and formatting. Initial analysis and formatting include segmenting the data. For example, assuming that the number of training devices in each group is d, and the number of devices performing loading in each group is t, the data is divided into d / t parts. If the amount of data cannot be divided by d / t, padding is adopted, that is, the data is padded with 0 so that it can be divided by d / t, and the data range to be received by each device is recorded, and the padded part is discarded after the data reception is completed.

[0159] 3. Data distribution;

[0160] After the initial loading and preprocessing, the scatter communication operator of PyTorch is used to distribute the data from the loading device to all training devices in a device-to-device (D2D) manner. Each device receives the model and optimizer data segments related only to its training task. At this time, the model and optimizer data segments are one-tenth of the total model data training devices, which greatly optimizes the data flow of the overall network and reduces bandwidth requirements.

[0161] 4. Synchronization and consistency check;

[0162] After data distribution is completed, a synchronization mechanism is implemented to ensure that the data on all devices is consistent. A flag is used as a marker, with an initial value of 0. After data distribution and reception are completed, it is set to 1, and the All-reduce algorithm is used to synchronize parameters across devices. If the flag obtained after all-reduce is the number of devices, it means that all devices have completed data distribution and reception. At this time, the subsequent steps are allowed to be executed, thereby ensuring that each step of model training can be carried out in a data-consistent state.

[0163] 5. Integrity verification and checking;

[0164] Use PyTorch's gather communication operator to collect parameter data from all training devices and return it to the master device or designated verification node. Here, the system will perform a detailed data integrity check to compare the data of each device with the original data to ensure that no errors or damage occurred during data transmission or processing.

[0165] 6. Fault detection and recovery;

[0166] An advanced fault detection mechanism is introduced to automatically identify and respond to any abnormal situation during data loading or distribution. For example, if a certain segment of model and optimizer data is found to be inconsistent or damaged with the original data during integrity verification and checking, the system will automatically reload the required data segment from the data source or cache, repeat the data distribution and verification steps, and ensure the robustness of the system. If the flag signal is not the device number for a long time during the synchronization and consistency check, the I / O operation of the corresponding device will be re-executed to ensure the continuity of training.

[0167] 7. Performance evaluation and optimization;

[0168] Since the system cannot guarantee stability during a large-scale distributed training process, a training process usually requires multiple checkpoint loading processes. After each round of checkpoint loading, collect and analyze system performance data to evaluate the implementation effect. Adjust the parameters of the evaluation model based on the performance results, optimize the data loading and distribution strategies to adapt to training needs of different scales and complexities, and improve the overall efficiency and response speed of the system. The specific rules are: if the I / O bandwidth of the first three checkpoint loadings is lower than the full bandwidth, randomly adjust the number of devices participating in the loading of each group of devices, and then fit the quadratic function of these three device quantities to the I / O bandwidth, and select the theoretically optimal number of devices to participate in the next checkpoint loading, and continue to collect I / O performance data and continue to iterate.

[0169] The disclosed embodiments demonstrate how to effectively apply the technical solution of the present invention in a large-scale distributed training environment with thousands of computing nodes, thereby significantly improving data processing efficiency and reducing operating costs by reducing unnecessary data transmission, optimizing resource allocation, and enhancing system robustness.

[0170] The technical solution of the disclosed embodiment brings the following significant beneficial effects by optimizing the checkpoint loading process in large-scale pre-training:

[0171] 1. Reduce bandwidth consumption: By performing initial data loading on limited devices and accurately distributing the required data using device-to-device communication, the overall network bandwidth requirements are significantly reduced. This is especially important for expensive data centers or cloud infrastructure, which can directly reduce operating costs.

[0172] 2. Improve data loading efficiency: The disclosed embodiment significantly improves data processing efficiency by precisely controlling data loading and distribution. Only necessary data is loaded onto relevant devices, reducing data transmission time in the network and waiting time on the device, thereby speeding up training startup.

[0173] 3. Optimize resource utilization: Concentrate data loading on a few devices and distribute them efficiently, so that the computing and storage resources of each device are more reasonably utilized. This not only improves the efficiency of a single device, but also improves the resource utilization of the entire training network.

[0174] 4. Improve system stability: By implementing advanced synchronization and verification mechanisms, the disclosed embodiments ensure the consistency and integrity of data on all devices. This stable data environment reduces possible errors during training and improves the reliability of model training.

[0175] 5. Enhance system scalability: The solution design of the disclosed embodiment takes into account scalability from small to large scales, and can flexibly adjust the data loading and distribution strategies according to different training scales. This allows the same technical solution to be applied to training tasks of different scales without requiring major structural adjustments.

[0176] 6. Promote green computing: reducing unnecessary data transmission and optimizing the use of computing devices not only reduces energy consumption, but also complies with the concept of green computing. In the current context of increasing requirements for environmental protection and energy efficiency, the embodiments of the present disclosure provide an environmentally friendly solution.

[0177] The disclosed embodiment can also be further optimized through the following scheme:

[0178] 1. A possible alternative is to optimize the distributed file system to support large-scale data operations more efficiently. By improving the data transmission and storage mechanism of the file system, the time for data loading and synchronization can be reduced. This approach relies on architectural improvements to the underlying file system and may require large system-level modifications, which usually requires huge time and manpower costs.

[0179] 2. Another solution is to apply more efficient data compression technology to reduce the volume of transmitted data. Although this can significantly reduce bandwidth usage, the compression and decompression process may add additional processing delays, increase the demand for computing resources accordingly, and reduce the accuracy of the model.

[0180] It should be noted that the embodiment of the present disclosure has the following protection points:

[0181] 1. Dynamic device selection algorithm: A unique evaluation model is used to dynamically determine the number of devices involved in the initial data loading. The evaluation model optimizes the selection of devices based on network bandwidth, device computing power, and pre-training task size to ensure the efficiency and economy of data loading. This is the core of the disclosed embodiment, which significantly optimizes the overall training process through intelligent resource allocation.

[0182] 2. Segmented data loading and distribution mechanism: The disclosed embodiment proposes an efficient data distribution strategy from device to device after preloading the full data set on limited devices. This method uses the scatter communication operator to accurately distribute data to the devices that need it, thereby reducing redundant data transmission and unnecessary bandwidth consumption.

[0183] 3. Synchronization and consistency check process: The disclosed embodiment includes a comprehensive data synchronization and consistency check mechanism to ensure that all devices have a consistent and complete data set before starting training. This not only improves the reliability of training, but also reduces errors and the need for retraining due to inconsistent data.

[0184] 4. Integrity verification and checking system: Data integrity verification using the gather communication operator is a key step to ensure that data has not been tampered with or incorrectly loaded during the distribution process. This system provides additional security for model training and ensures the accuracy and credibility of the data.

[0185] 5. Fault detection and recovery strategy: The fault detection and automatic recovery mechanism in the disclosed embodiment provides a high degree of fault tolerance for the training system. During data loading or distribution, the system can automatically identify faults and re-execute necessary steps to ensure the continuity of training and the integrity of data.

[0186] 6. Scalability and adaptability design: The technical solution of the disclosed embodiment is highly scalable and can flexibly adapt to different training environments from small to large scale. This design can work effectively under a variety of hardware configurations and network conditions, greatly improving its application scope and practicality.

[0187] Based on the foregoing embodiments, the embodiments of the present disclosure provide a model data loading device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0188] Figure 6 A schematic diagram of the structure of a model data loading device provided in an embodiment of the present disclosure Figure 1 ,like Figure 6 As shown, the model data loading device 600 includes: a first determining module 610 and a loading module 620, wherein:

[0189] A first determination module 610 is used to determine a second device that participates in data loading from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices, and the model data to be processed; the data loading capability is used to reflect the processing efficiency and the network bandwidth occupied by the first device when loading data; the model data to be processed is the relevant data of the model state saved at a specific time point during the training process;

[0190] The loading module 620 is used to load the model data to be processed into the second device, so as to distribute the model data to be processed to the devices participating in the model training in the distributed model training system through the second device.

[0191] In some embodiments, the first determination module 610 is further used to: group the first devices based on the topological structure between the first devices to obtain multiple device groups; and determine the second device from the multiple device groups based on the model data to be processed and the data loading capacity of each device in the device group.

[0192] In some embodiments, the first determination module 610 is also used to: determine the number of files storing the model data to be processed; determine the first number of devices in the device group participating in data loading based on the number of files and the number of device groups; sort the training devices in the device group based on the data loading capability of each device in the device group; and use the first number of training devices ranked first in the device group as the second devices.

[0193] In some embodiments, the first determination module 610 is further used to: determine the performance data of the distributed model training system after each round of data loading is completed; and adjust the devices in the device group that participate in data loading based on the performance data.

[0194] In some embodiments, the first determination module 610 is further used to: determine whether the network bandwidth in the performance data within the target round is lower than a bandwidth threshold; and when the network bandwidth within the target round is lower than the bandwidth threshold, adjust the first quantity based on the number of devices participating in data loading within the target round and the network bandwidth.

[0195] In some embodiments, the first determination module 610 is further used to: perform digital fitting on the number of devices participating in data loading in the target round and the network bandwidth to determine the optimal number of devices participating in data loading; and adjust the first number to the optimal number.

[0196] In some embodiments, the model data loading device 600 further includes:

[0197] A detection module is used to detect the integrity of the model data to be processed based on the first data to be detected sent back by the third device participating in the model training; to detect the synchronization of multiple third devices based on the second data to be detected sent back by the third device; if both the integrity and the synchronization are detected successfully, determine that there is no abnormality in the distribution process of the model data to be processed; if at least one of the integrity and the synchronization fails to be detected, determine that there is an abnormality in the distribution process of the model data to be processed.

[0198] In some embodiments, the detection module is also used to: receive the model data to be verified transmitted by the third device; the first data to be detected includes the model data to be verified; determine whether the model data to be verified matches the model data to be processed; if any of the model data to be verified does not match the model data to be processed, determine that the integrity check fails; if the model data to be verified matches the model data to be processed, determine that the integrity check is successful.

[0199] In some embodiments, the detection module is also used to: obtain an identifier corresponding to the third device; the second data to be detected includes the identifier; the identifier is used to indicate whether the data block has been received by the corresponding third device; when the number of the identifiers is equal to the number of the third devices and the identifiers are all first values, determine that the synchronization detection is successful; the first value indicates that the data block has been received by the corresponding third device; when the number of the identifiers is not equal to the number of the third devices, and / or any of the identifiers is a second value, determine that the distribution of the model data to be processed has failed; the second value indicates that the data block has not been received by the corresponding third device.

[0200] Based on the foregoing embodiments, the embodiments of the present disclosure provide another model data loading device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0201] Figure 7A schematic diagram of the structure of a model data loading device provided in an embodiment of the present disclosure Figure 2 ,like Figure 7 As shown, the model data loading device 700 includes: a second determination module 710, a division module 720 and a distribution module 730, wherein:

[0202] The second determination module 710 is used to determine the fourth device participating in model training and the fifth device participating in data loading in the device group where the second device is located after the second device participating in data loading in the distributed model training system obtains the model data to be processed; the model data to be processed is the relevant data of the model state saved at a specific time point in the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed;

[0203] A partitioning module 720, configured to partition the model data to be processed based on the number of the fourth devices and the number of the fifth devices to obtain data blocks related to the training task of the fourth device;

[0204] The distribution module 730 is used to distribute each of the data blocks to the corresponding fourth device, so as to distribute the model data to be processed to the devices participating in the model training in the distributed model training system.

[0205] In some embodiments, the partitioning module 720 is also used to: determine the number of partitions of the model data to be processed based on the number of the fourth devices and the number of the fifth devices; and partition the model data to be processed based on the number of partitions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

[0206] In some embodiments, the partitioning module 720 is also used to: determine whether the total data volume of the model data to be processed is divisible by the number of partitions; if the total data volume is divisible by the number of partitions, the model data to be processed is partitioned based on the number of partitions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

[0207] In some embodiments, the division module 720 is also used for: when the total data volume cannot be divided by the division number, using preset characters to pad the model data to be processed to obtain the padded model data to be processed; the data volume of the padded model data to be processed can be divided by the division number, and the preset characters will not affect the specific content of the model data to be processed; based on the division number and the data requirements of the training task of the fourth device, the padded model data to be processed is divided to obtain data blocks related to the training task of the fourth device.

[0208] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0209] It should be noted that in the embodiments of the present disclosure, if the above-mentioned model data loading method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0210] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0211] The embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0212] An embodiment of the present disclosure provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0213] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0214] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product disclosed in the present invention, please refer to the description of the method embodiment disclosed in the present invention for understanding.

[0215] It should be noted that Figure 8 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure, such as Figure 8 As shown, the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802 and a memory 803, wherein:

[0216] Processor 801 generally controls the overall operation of computer device 800 .

[0217] The communication interface 802 enables the computer device to communicate with other terminals or servers through a network.

[0218] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the computer device 800 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 801, the communication interface 802, and the memory 803 through the bus 804.

[0219] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0220] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0221] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0222] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0223] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0224] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0225] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0226] The above description is only an implementation mode of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure.

Claims

1. A model data loading method, characterized in that: The model data loading method comprises: Based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed, a second device participating in data loading is determined from the first device; the data loading capability is used to reflect the processing efficiency and the network bandwidth occupied by the first device when loading data; the model data to be processed is the relevant data of the model state saved at a specific time point during the training process; The model data to be processed is loaded into the second device, so as to distribute the model data to be processed to the devices participating in the model training in the distributed model training system through the second device.

2. The model data loading method according to claim 1, characterized in that: The method of determining a second device that participates in data loading from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices, and the model data to be processed includes: Grouping the first devices based on the topological structure between the first devices to obtain multiple device groups; The second device is determined from a plurality of the device groups based on the model data to be processed and a data loading capability of each device in the device group.

3. The model data loading method according to claim 2, characterized in that: The determining the second device from a plurality of the device groups based on the model data to be processed and the data loading capability of each device in the device group comprises: Determining the number of files storing the model data to be processed; Based on the number of the files and the number of the device groups, determining a first number of devices in the device group that participate in data loading; sorting the training devices in the device group based on the data loading capability of each device in the device group; The first number of training devices ranked first in the device group are used as the second devices.

4. The model data loading method according to claim 3, characterized in that: The model data loading method further includes: After each round of data loading is completed, determining performance data of the distributed model training system; Based on the performance data, devices in the device group that participate in data loading are adjusted.

5. The model data loading method according to claim 4, characterized in that: The step of adjusting the devices in the device group that participate in data loading based on the performance data includes: Determining whether the network bandwidth in the performance data in the target round is lower than a bandwidth threshold; When the network bandwidth in the target round is lower than the bandwidth threshold, the first number is adjusted based on the number of devices participating in data loading in the target round and the network bandwidth.

6. The model data loading method according to claim 5, characterized in that: The adjusting the first number based on the number of devices participating in data loading in the target round and the network bandwidth includes: Performing digital fitting on the number of devices participating in data loading in the target round and the network bandwidth to determine the optimal number of devices participating in data loading; The first number is adjusted to the optimal number.

7. The model data loading method according to any one of claims 1 to 6, characterized in that: The model data loading method further includes: Detecting the integrity of the model data to be processed based on the first data to be detected sent back by the third device participating in the model training; Detecting the synchronization of the plurality of third devices based on the second data to be detected returned by the third device; In the case where both the integrity and the synchronization are detected successfully, determining that no abnormality occurs in the distribution process of the model data to be processed; In the case where at least one of the integrity and the synchronization fails to be detected, it is determined that an abnormality exists in the distribution process of the model data to be processed.

8. The model data loading method according to claim 7, characterized in that: The detecting the integrity of the model data to be processed based on the first data to be detected returned by the third device participating in the model training includes: receiving the model data to be verified transmitted by the third device; the first data to be detected includes the model data to be verified; Determining whether the model data to be verified matches the model data to be processed; In the event that any of the model data to be verified does not match the model data to be processed, determining that the integrity check fails; In a case where the model data to be verified matches the model data to be processed, it is determined that the integrity check is successful.

9. The model data loading method according to claim 7, characterized in that: The detecting the synchronization of the plurality of third devices based on the second data to be detected returned by the third device includes: Acquire an identifier corresponding to the third device; the second data to be detected includes the identifier; the identifier is used to indicate whether the data block has been received by the corresponding third device; In a case where the number of the identifiers is equal to the number of the third devices and the identifiers are all first values, determining that the synchronization detection is successful; the first value indicates that the data block has been received by the corresponding third device; When the number of the identifiers is not equal to the number of the third devices, and / or any of the identifiers is a second numerical value, it is determined that the distribution of the model data to be processed has failed; the second numerical value indicates that the data block has not been received by the corresponding third device.

10. A model data loading method, characterized in that: The model data loading method comprises: After the second device participating in data loading in the distributed model training system obtains the model data to be processed, the fourth device participating in model training and the fifth device participating in data loading in the device group where the second device is located are determined; the model data to be processed is the relevant data of the model state saved at a specific time point in the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed; Dividing the model data to be processed based on the number of the fourth devices and the number of the fifth devices to obtain data blocks related to the training task of the fourth device; Distribute each of the data blocks to the corresponding fourth device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system.

11. The model data loading method according to claim 10, characterized in that: The dividing the model data to be processed based on the number of the fourth devices and the number of the fifth devices to obtain data blocks related to the training task of the fourth device includes: Determining the number of divisions of the to-be-processed model data based on the number of the fourth devices and the number of the fifth devices; The model data to be processed is divided based on the number of divisions and the data requirement of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

12. The model data loading method according to claim 11, characterized in that: The dividing the model data to be processed based on the number of divisions and the data requirement of the training task of the fourth device to obtain data blocks related to the training task of the fourth device includes: Determine whether the total data volume of the model data to be processed is divisible by the number of divisions; In the case that the total data volume is divisible by the number of divisions, the model data to be processed is divided based on the number of divisions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

13. The model data loading method according to claim 12, characterized in that: The model data loading method further includes: In the case that the total data volume cannot be divided by the number of divisions, the model data to be processed are padded with preset characters to obtain the padded model data to be processed; the data volume of the padded model data to be processed can be divided by the number of divisions, and the preset characters will not affect the specific content of the model data to be processed; The completed model data to be processed is divided based on the number of divisions and the data requirements of the training task of the fourth device to obtain data blocks related to the training task of the fourth device.

14. A model data loading device, characterized in that: The model data loading device comprises: A first determination module is used to determine a second device that participates in data loading from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices, and the model data to be processed; the data loading capability is used to reflect the processing efficiency and the network bandwidth occupied by the first device when loading data; the model data to be processed is the relevant data of the model state saved at a specific time point during the training process; A loading module is used to load the model data to be processed into the second device, so as to distribute the model data to be processed to the devices participating in model training in the distributed model training system through the second device.

15. A model data loading device, characterized in that: The model data loading device comprises: A second determination module is used to determine, after the second device participating in data loading in the distributed model training system obtains the model data to be processed, a fourth device participating in model training and a fifth device participating in data loading in the device group where the second device is located; the model data to be processed is relevant data of the model state saved at a specific time point during the training process; the second device is determined from the first device based on the data loading capability of the first device included in the distributed model training system, the topological structure between the first devices and the model data to be processed; a partitioning module, configured to partition the model data to be processed based on the number of the fourth devices and the number of the fifth devices, to obtain data blocks related to the training task of the fourth device; A distribution module is used to distribute each of the data blocks to the corresponding fourth device, so as to distribute the model data to be processed to the devices participating in the model training in the distributed model training system.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 or 10 to 13 are implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 or 10 to 13 are implemented.

18. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, the steps of the method according to any one of claims 1 to 9 or 10 to 13 are implemented.

Images