A Distributed Training Acceleration System Based on Distributed Storage

Abstract

This invention relates to a distributed training acceleration system based on distributed storage, comprising a data orchestration unit that caches the dataset according to a first caching level rule, performs security isolation on the dataset, and uses the namespace of an open-source container orchestration engine to isolate resources; and a data hierarchical caching acceleration unit that caches the dataset according to a second caching level rule, performs dynamic preloading on the dataset, and accelerates the loading of distributed training data, thereby effectively improving the speed of distributed training based on distributed storage.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and in particular to a distributed training acceleration system based on distributed storage. Background Technology

[0002] Medical artificial intelligence (AI) model training is one of the main functions of a medical AI platform.

[0003] Most existing deep learning training platforms are built on the open-source container orchestration engine Kubernetes cluster. Therefore, distributed training on these platforms faces challenges such as high I / O communication pressure and complex data management, resulting in slow distributed training speeds and I / O bottlenecks impacting resource utilization. When the dataset is in the cloud, the speed of distributed training is limited by the communication capabilities between the network and the cluster hosts; when the dataset is local, the speed is limited by network latency caused by NFS mounting and disk-level I / O reads.

[0004] In summary, existing methods, regardless of whether the dataset is in the cloud or locally, suffer from significant I / O communication bottlenecks, resulting in slow distributed training speeds. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a distributed training acceleration system based on distributed storage, thereby improving the speed of distributed training. To achieve the above objectives, the present invention provides a distributed training acceleration system based on distributed storage, comprising: The data orchestration unit performs caching processing on the dataset according to the first caching level rules, performs security isolation processing on the dataset, and uses the namespace of the open-source container orchestration engine to isolate resources. The data hierarchical caching acceleration unit performs caching processing on the dataset according to the second caching level rules, performs dynamic preloading processing on the dataset, and accelerates the loading of distributed training data.

[0006] Preferably, the data orchestration unit is configured as follows: Data storage involves caching the data in a distributed storage cluster (Ceph cluster) according to the first cache level rules. Configure the open-source container orchestration engine cluster by attaching the master node and worker nodes of the open-source container orchestration engine cluster to the distributed storage cluster; Data mounting involves mounting the data to the underlying file storage of a preset data orchestration platform using a pre-written configuration file; wherein the configuration file includes first cache level rules and custom resources for the created dataset; Persistent volume and persistent volume declaration generation: Persistent volume and persistent volume declaration are generated in the open-source container orchestration engine cluster. Dynamic adjustment: Based on the first cache level rule, the cache level of the first dataset is dynamically adjusted according to the usage frequency of the first dataset.

[0007] Preferably, the dynamic adjustment specifically includes: Calculate the first usage frequency of each first dataset within the first time period; Adjust the cache level of the first dataset based on the frequency of use of each first dataset.

[0008] More preferably, adjusting the cache level of the first dataset according to the first usage frequency of each first dataset specifically includes: Adjust the cache level of the first dataset whose usage frequency is greater than the first preset frequency to the first cache level; The cache level of the first dataset whose usage frequency is greater than the second preset frequency and less than or equal to the first preset frequency is adjusted to the second cache level; The cache level of the first dataset, whose usage frequency is less than or equal to the second preset frequency, is adjusted to the third cache level; Wherein, the first preset frequency is greater than the second preset frequency, and the levels of the first cache level, the second cache level, and the third cache level gradually decrease.

[0009] Preferably, the first cache level rules include memory level, disk level, and hard disk level.

[0010] More preferably, the first cache level rule further includes a first preset value corresponding to the memory level, a second preset value corresponding to the disk level, and a third preset value corresponding to the hard disk level.

[0011] Preferably, the second cache level rule includes a memory level and a fourth preset value corresponding to the memory level, a disk level and a fifth preset value corresponding to the disk level, and a hard disk level and a sixth preset value corresponding to the hard disk level.

[0012] Preferably, the data hierarchical caching acceleration unit is configured as follows: Data slice parameter processing, receiving relevant parameters from R computing nodes (workers) of distributed training using a preset custom resource type, and performing calculation processing based on the relevant parameters of the distributed training computing nodes to obtain the nearest N data subsets; Heat definition processing, defining the cache level for each of the N data subsets according to the chronological order of the N data subsets and the size of each cache level in the second cache level rule; Hierarchical caching, hierarchically caching the N data subsets using a preset data orchestration platform according to the cache level of each of the N data subsets; Selection of random seeds, selecting different or fixed random seeds according to the epoch; Data subset loading and updating, loading each of the N data subsets in order from the highest to the lowest cache priority, and recalculating the data subsets to be cached when the lowest priority cache is emptied.

[0013] Further preferably, the relevant parameters of the distributed training computing nodes include the data sharding random seed, the unique identifier (rank value) of the computing node (worker), and the current training epoch (the iteration number is iteration). The calculation processing based on the relevant parameters of the distributed training computing nodes to obtain the nearest N data subsets specifically includes: According to R computing nodes (workers), the total number of samples in the dataset is M, and a 1×M dataset is randomly shuffled and generated for each epoch; where R is a positive integer; According to the dataset size B and the data subset size S, the total number of data subsets is calculated as: s = M ÷ S where B is a positive integer, S is a positive integer, and s is a positive integer; According to the current training epoch sequence number Iter and the unique identifier r assigned to the current computing node (Worker), the sequence number SubBatch_idx of the current data subset is calculated as: SubBatch_idx = Iter×R + r where 0 ≤ r<R and SubBatch_idx is a positive integer; Obtain the epochs corresponding to N SubBatch_idx from Iter to Iter + N + 1 to obtain the nearest N data subsets.

[0014] Further preferably, the data subset loading and updating specifically includes: Calculate the sequence number of the current data subset assigned to the current computing node based on the iteration number of the round, and store the corresponding N data subsets into the cache in order; As training progresses, Iter continuously increases, taking a subset of data from the high-priority cache and replenishing the high-priority cache with a subset of data from the low-priority cache. When the data in the lowest priority cache is cleared, the subset of data to be cached is recalculated and added to the low priority cache.

[0015] This invention provides a distributed training acceleration system based on distributed storage. It incorporates a data orchestration unit that caches the distributed storage dataset at three levels: memory, disk, and hard disk. It utilizes the namespaces of the open-source container orchestration engine Kubernetes to isolate resources and employs a hierarchical data caching acceleration unit. Based on this unit, the pods corresponding to the distributed training compute nodes can mount volumes with multi-level caching capabilities of a pre-defined custom resource type. This ensures that each compute node retrieves a subset of hot data from the high-priority cache for each training iteration, maximizing data loading speed and thus guaranteeing that distributed training efficiency reaches the system's limit, significantly improving training speed. With this design, distributed training tasks based on Kubernetes require no modification; the pre-defined custom resource type `dataload` automatically implements data caching and preloading, and algorithm developers incur no code modification costs. This distributed training acceleration system based on distributed storage solves the problem of distributed training data I / O being limited by cluster storage performance. Attached Figure Description

[0016] Figure 1 A system block diagram of a distributed training acceleration system based on distributed storage provided in an embodiment of the present invention; Figure 2 A system block diagram of a distributed training acceleration system based on distributed storage, comprising a configuration data orchestration unit and a configuration data hierarchical caching acceleration unit, provided in an embodiment of the present invention; Figure 3 A flowchart for data slicing parameter processing provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the data subset loading and updating process provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0019] The distributed training acceleration system based on distributed storage provided in this invention applies hierarchical caching technology to distributed training technology, which changes the data loading method of distributed training in the prior art and solves the problem that distributed training data I / O is limited by the performance of cluster storage.

[0020] Figure 1 A system block diagram of a distributed training acceleration system based on distributed storage provided in an embodiment of the present invention is shown below. Figure 1 The technical solution of the present invention will be described with reference to specific embodiments.

[0021] like Figure 1 As shown in the figure, an embodiment of the present invention provides a distributed training acceleration system 10000 based on distributed storage, comprising: a data orchestration unit 10001 and a data hierarchical caching acceleration unit 10002. Wherein, Data orchestration unit 10001 performs caching and security isolation of the dataset according to the first caching level rules, and uses the namespace of the open-source container orchestration engine to isolate resources.

[0022] Specifically, in this embodiment of the invention, the first cache level rule is a cache level divided according to cache priority, including three cache levels: memory level, disk level, and hard disk level. The data orchestration unit 10001 caches the datasets in memory, disk, and hard disk respectively according to their priority from high to low, and uses the namespace of the open-source container orchestration engine to isolate resources and securely isolate the stored data. In this embodiment of the invention, the open-source container orchestration engine is selected from existing technologies; specifically, Kubernetes is selected as the open-source container orchestration engine. Kubernetes, or K8s for short, is a commonly used open-source container orchestration engine in existing technologies. It simplifies the deployment, scaling, and management of containerized applications. Its core functions include: automated management, supporting automated deployment, scaling, rolling updates, and rollback of applications, and continuous maintenance of desired states through declarative APIs; service discovery and load balancing, with built-in DNS service and Ingress controller to achieve automatic routing and traffic distribution for inter-service communication; and resource scheduling, allocating optimal nodes to the smallest deployable unit, Pod, through its default scheduler, Kube-scheduler component, considering constraints such as resource requirements and affinity rules.

[0023] Figure 2 A system block diagram of a distributed training acceleration system based on distributed storage, comprising a configuration data orchestration unit and a configuration data hierarchical caching acceleration unit, is provided in an embodiment of the present invention. Figure 2 As shown, in a preferred embodiment of the present invention, the data orchestration unit 10001 is configured to perform the following processes: data storage 11001, open-source container orchestration engine configuration 12001, data mounting 13001, persistent volume and persistent volume declaration generation 14001, and dynamic adjustment 15001. Each process will now be described in detail.

[0024] Data storage 11001 caches the data in a distributed storage cluster according to the first cache level rules. In this embodiment of the invention, the distributed storage cluster is selected from existing technologies. For example, a Ceph cluster is selected as the distributed storage cluster of this invention. Ceph is specifically an open-source distributed storage system that supports block storage, file storage, and object storage. Data is cached in descending order of priority into the storage units of each cache level specified by the first cache level rules. For example, data is cached in memory, disk, and hard disk respectively in descending order of priority.

[0025] The open-source container orchestration engine cluster configuration 12001 mounts the master and worker nodes of the open-source container orchestration engine cluster to a distributed storage cluster. In this embodiment of the invention, the compute nodes of the Kubernetes cluster, namely the master node and worker nodes, need to be mounted to the Ceph cluster to ensure low-latency data sharing.

[0026] Data mounting 13001 mounts data to the underlying file storage of a pre-written data orchestration platform (Alluxio) using a pre-written configuration file. In this embodiment, the pre-written data orchestration platform is selected from existing technical solutions; for example, in a preferred embodiment, Alluxio is chosen as the pre-written data orchestration platform. In this embodiment, the configuration file includes first cache level rules and created custom dataset resources, i.e., custom dataset resources. This can be achieved by specifying Ceph storage volumes in the Kubernetes configuration file. That is, by pre-writing a YAML file, data stored in the Ceph cluster is mounted to Alluxio's underlying file storage (UFS).

[0027] Persistent Volume and Persistent Volume Claim Generation 14001: Persistent volumes and persistent volume claims are generated in an open-source container orchestration engine cluster. In this embodiment, persistent volumes (PVs) and persistent volume claims (PVCs) are generated in a Kubernetes cluster for the purpose of mounting these PVs and PVCs to containers during subsequent containerized distributed training.

[0028] The dynamic adjustment of cache level 15001 is based on a first cache level rule, dynamically adjusting the cache level of the first dataset according to its usage frequency. In a preferred embodiment of the invention, the dynamic adjustment specifically includes: first, calculating the first usage frequency of each first dataset within a first time period; second, adjusting the cache level of the first dataset according to its first usage frequency. Specifically, adjusting the cache level of the first dataset according to its first usage frequency includes: adjusting the cache level of first datasets with a first usage frequency greater than a first preset frequency to the first cache level; adjusting the cache level of first datasets with a first usage frequency greater than a second preset frequency but less than or equal to the first preset frequency to the second cache level; and adjusting the cache level of first datasets with a first usage frequency less than or equal to the second preset frequency to the third cache level; wherein the first preset frequency is greater than the second preset frequency, and the levels of the first, second, and third cache levels gradually decrease. In a preferred embodiment of the invention, the first cache level is the memory level, the second cache level is the disk level, and the third cache level is the hard disk level.

[0029] In a preferred embodiment of the present invention, the first cache level rule further includes a first preset value corresponding to the memory level, a second preset value corresponding to the disk level, and a third preset value corresponding to the hard disk level. The first, second, and third preset values ​​are all dynamically configurable values ​​representing the size of the cache space. It should be noted that in this embodiment of the present invention, the cache priority is highest for memory, followed by disk (typically referring to an SSD all-flash array), and lastly for hard disk (typically referring to a traditional SATA mechanical hard disk).

[0030] The data hierarchical caching acceleration unit 10002 performs caching processing on the dataset according to the second caching level rules, performs dynamic preloading processing on the dataset, and accelerates the loading of distributed training data.

[0031] Specifically, in a preferred embodiment of the present invention, the second cache level rules include a memory level and a fourth preset value corresponding to the memory level, a disk level and a fifth preset value corresponding to the disk level, and a hard disk level and a sixth preset value corresponding to the hard disk level. The fourth, fifth, and sixth preset values ​​are dynamically configurable values ​​representing the size of the cache space or storage space. It should be noted that in this embodiment of the present invention, the cache priority is highest for memory, followed by disk (typically referring to an SSD all-flash array), and lastly for hard disks (typically referring to a traditional SATA mechanical hard disk).

[0032] like Figure 2As shown in the figure, in the preferred solution of the embodiment of the present invention, the data hierarchical caching acceleration unit 10002 is configured to perform the following processing, including: data slicing parameter processing 11002, heat definition processing 12002, hierarchical caching 13002, selection of random seeds 14002, and data subset loading and updating 15002. Next, each processing will be introduced in detail.

[0033] For the data slicing parameter processing 11002, relevant parameters from R computing nodes (workers) of distributed training are received using a preset custom resource type, and calculation processing is performed based on the relevant parameters of the distributed training computing nodes to obtain the nearest N data subsets. Figure 3 This is a flowchart of a data slicing parameter processing provided by an embodiment of the present invention. As Figure 3 shown, in the solution provided by the embodiment of the present invention, the relevant parameters of the distributed training computing nodes include the data splitting random seed, the unique identifier (rank value) of the computing node (worker), and the current training round (iteration). Performing calculation processing based on the relevant parameters of the distributed training computing nodes to obtain the nearest N data subsets specifically includes the following steps: Step S110: Based on R computing nodes (workers), the total number of samples in the dataset is M, and a 1×M dataset is randomly shuffled and generated for each round (epoch); where R is a positive integer; Step S120: Calculate the total number of data subsets according to the dataset size B and the data subset size S as: s = M ÷ S (Equation 1) where B is a positive integer, S is a positive integer, and s is a positive integer; Step S130: Perform calculation processing according to the current training round number Iter and the unique identifier r assigned to the current computing node (Worker) to obtain the sequence number SubBatch_idx of the current data subset as: SubBatch_idx = Iter×R + r (Equation 2) where 0 ≤ r < R, and SubBatch_idx is a positive integer; Step S140: Obtain the rounds corresponding to N SubBatch_idx from Iter to Iter + N + 1 to obtain the nearest N data subsets.

[0034] The following example illustrates the above steps. Assume there are R workers, the total sample size in the dataset is M, and each epoch randomly generates a 1×M dataset. Batchsize=B, subBatchSize=S, then there are a total of s=M / S subBatches. We can then calculate the iteration number (Iter) for each iteration, and assign the subBatch number (SubBatch_idx = Iter×R + r) to the current worker with a rank of r. Taking the next N iterations, with numbers ranging from Iter to Iter+N-1, we obtain the indices of the nearest N subBatches, which are the indices of the N sub-datasets.

[0035] The popularity definition process 12002 defines a cache level for each of the N data subsets based on their temporal order and the space size of each cache level in the second cache level rule. In other words, these N data subsets are defined into three popularity levels according to their temporal order and the space size of the three caches. Then, based on the popularity level of each data subset, they are cached in memory, disk, or hard drive as specified in the second cache rule.

[0036] Tiered caching 13002 involves caching N data subsets according to the caching level of each subset using a preset data orchestration platform. In this embodiment, the preset data orchestration platform is Alluxio. That is, Alluxio is used to cache the N data subsets in memory, disk, and hard disk according to their priority from high to low.

[0037] The selection of random seeds (14002) involves choosing different or fixed random seeds based on the epoch. In distributed training, different random seeds are typically chosen for each epoch to provide better randomness. However, in some training scenarios, a fixed random seed is required. This solution supports the use of different or fixed random seeds.

[0038] Data subset loading and updating 15002: Load each of the N data subsets in descending order of cache priority, and recalculate the data subset to be cached when the lowest priority cache is cleared. Figure 4 The flowchart for loading and updating data subsets provided in the embodiments of the present invention is as follows: Figure 4 As shown, in a preferred embodiment of the present invention, the data subset loading and updating specifically includes the following steps: Step S210: Calculate the sequence number of the current data subset assigned to the current computing node according to the iteration number of the round, and store the corresponding N data subsets into the cache in order.

[0039] In step S220, as training progresses, Iter continuously increases, and data subsets are retrieved from the high-priority cache, while data subsets from the low-priority cache are added to the high-priority cache.

[0040] Step S230: When the data in the lowest priority cache is cleared, the subset of data to be cached is recalculated and added to the low priority cache.

[0041] In this embodiment of the invention, the Pod corresponding to the worker in the distributed training based on the data hierarchical caching acceleration unit 10002 can ensure that each worker's training runs from the highest priority cache (i.e., the memory cache) to obtain a subset of hot data. This ensures that the data loading speed is at the system's maximum speed, guaranteeing that the distributed training efficiency reaches the system's limit. However, the memory resources corresponding to hot datasets are limited, consuming more resources in the Kubernetes cluster. Therefore, higher-priority caches typically have smaller storage spaces. Specifically, when dynamically configuring the fourth, fifth, and sixth preset values, the fourth preset value can be configured as the minimum value among them, such that the fourth preset value < the fifth preset value < the sixth preset value.

[0042] This invention provides a distributed training acceleration system based on distributed storage. Through data orchestration, it performs three-level caching of data in distributed storage: memory, disk, and hard disk. It also uses Kubernetes namespaces to isolate resources and securely isolate stored data. Through hierarchical data caching technology, it achieves caching and dynamic preloading of datasets, solving the problem of low training efficiency caused by high I / O communication pressure and slow loading in distributed training. Furthermore, this invention eliminates the need for modifications to Kubernetes-based distributed training tasks, as the dataload resource can automatically perform data caching and preloading. This reduces code modification costs in algorithm development, improves development efficiency, and solves the problem of distributed training data I / O being limited by cluster storage performance.

[0043] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0044] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0045] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed training acceleration system based on distributed storage, characterized in that, The distributed training acceleration system includes: The data orchestration unit performs caching processing on the dataset according to the first caching level rules, performs security isolation processing on the dataset, and uses the namespace of the open-source container orchestration engine to isolate resources. The data hierarchical caching acceleration unit performs caching processing on the dataset according to the second caching level rules, performs dynamic preloading processing on the dataset, and accelerates the loading of distributed training data.

2. The distributed training acceleration system according to claim 1, characterized in that, The data orchestration unit is configured as follows: Data storage involves caching the data in a distributed storage cluster according to the first cache level rules; Configure the open-source container orchestration engine cluster by attaching the master node and worker nodes of the open-source container orchestration engine cluster to the distributed storage cluster; Data mounting involves mounting the data to the underlying file storage of a preset data orchestration platform using a pre-written configuration file; wherein the configuration file includes first cache level rules and custom resources for the created dataset; Persistent volume and persistent volume declaration generation: Persistent volume and persistent volume declaration are generated in the open-source container orchestration engine cluster. Dynamic adjustment: Based on the first cache level rule, the cache level of the first dataset is dynamically adjusted according to the usage frequency of the first dataset.

3. The distributed training acceleration system according to claim 1, characterized in that, The dynamic adjustment specifically includes: Calculate the first usage frequency of each first dataset within the first time period; Adjust the cache level of the first dataset based on the frequency of use of each first dataset.

4. The distributed training acceleration system according to claim 3, characterized in that, The specific steps of adjusting the cache level of the first dataset based on the first usage frequency of each first dataset include: Adjust the cache level of the first dataset whose usage frequency is greater than the first preset frequency to the first cache level; The cache level of the first dataset whose usage frequency is greater than the second preset frequency and less than or equal to the first preset frequency is adjusted to the second cache level; The cache level of the first dataset, whose usage frequency is less than or equal to the second preset frequency, is adjusted to the third cache level; Wherein, the first preset frequency is greater than the second preset frequency, and the levels of the first cache level, the second cache level, and the third cache level gradually decrease.

5. The distributed training acceleration system according to claim 1, characterized in that, The first cache level rules include memory level, disk level, and hard disk level.

6. The distributed training acceleration system according to claim 5, characterized in that, The first cache level rule also includes a first preset value corresponding to the memory level, a second preset value corresponding to the disk level, and a third preset value corresponding to the hard disk level.

7. The distributed training acceleration system according to claim 1, characterized in that, The second cache level rule includes a memory level and a fourth preset value corresponding to the memory level, a disk level and a fifth preset value corresponding to the disk level, and a hard disk level and a sixth preset value corresponding to the hard disk level.

8. The distributed training acceleration system according to claim 1, characterized in that, The data tiered caching acceleration unit is configured as follows: Data slicing parameter processing involves receiving relevant parameters from R computing nodes in a distributed training environment using a preset custom resource type, and performing calculations based on the relevant parameters of the distributed training computing nodes to obtain the most recent N data subsets. Heat definition processing, defining the cache level for each of the N data subsets respectively according to the chronological order of the N data subsets and the sizes of the spaces of each cache level in the second cache level rule; Hierarchical caching, hierarchically caching the N data subsets using a preset data orchestration platform according to the cache level of each of the N data subsets; Selection of random seeds, selecting different or fixed random seeds according to the round; Data subset loading and update, loading each of the N data subsets in order from the highest to the lowest cache priority, and recalculating the data subsets to be cached when the lowest priority cache is emptied.

9. The distributed training acceleration system according to claim 8, characterized in that, The relevant parameters of the distributed training computing node include the data sharding random seed, the unique identifier of the computing node, and the current training round. The calculation process to obtain the most recent N data subsets according to the relevant parameters of the distributed training computing node specifically includes: According to R computing nodes, the total number of samples in the dataset is M, and a 1×M dataset is randomly shuffled and generated in each round; where R is a positive integer; Calculating the total number of data subsets according to the dataset size B and the data subset size S as: s = M ÷ S where B is a positive integer, S is a positive integer, and s is a positive integer; Calculating through the current training round number Iter and the unique identifier r assigned to the current computing node to obtain the serial number SubBatch_idx of the current data subset as: SubBatch_idx = Iter×R + r where 0 ≤ r<R and SubBatch_idx is a positive integer; Obtaining the rounds corresponding to the N SubBatch_idx from Iter to Iter + N + 1 to obtain the most recent N data subsets.

10. The distributed training acceleration system according to claim 9, characterized in that, The data subset loading and update specifically includes: Calculating the serial number of the current data subset assigned to the current computing node according to the iteration number of the round, and storing the corresponding N data subsets into the cache in order; As the training progresses, Iter continuously increases, taking out data subsets from the cache with the highest priority, and the data subsets in the cache with the lowest priority are replenished to the cache with the highest priority; When the data in the lowest priority cache is emptied, recalculate the data subsets to be cached and replenish them to the lowest priority cache.

Images