A distributed GPU cluster storage resource balancing method based on federated learning

Abstract

The application discloses a distributed GPU cluster storage resource balancing method based on federated learning, relates to the technical field of distributed computing and storage resource management, and comprises the following steps: collecting the read-write operation records of model parameter data blocks, gradient data blocks and training sample data blocks of each GPU computing node and a central aggregation server during the execution of a federated learning training round, generating real-time access logs, and calculating the access frequency distribution of each data block based on the current training round; the distributed GPU cluster storage resource balancing method based on federated learning realizes dynamic balanced allocation of storage resources, reduces the overall time consumption of the federated learning training round, and improves the training performance and system stability in the distributed GPU cluster environment.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and storage resource management technology, specifically to a method for balancing storage resources in a distributed GPU cluster based on federated learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, federated learning, as a distributed machine learning technology that enables multi-party collaborative modeling while ensuring data privacy, has been widely applied in scenarios such as edge computing, medical data modeling, and financial risk control. A typical federated learning system usually includes multiple computing nodes and a central aggregation server. After each participating node completes model training locally, it uploads the model parameters or gradient information to the central server for aggregation, and repeatedly executes the model distribution and update process in multiple training rounds.

[0003] In practical engineering deployments, federated learning typically runs in a distributed GPU cluster environment to meet the demands of large-scale model training. This environment includes multiple GPU computing nodes, each configured with local storage resources. The system also includes a shared storage layer and a cache storage layer to store model parameters, gradient data, and training sample data. As training epochs progress, the access frequency of different types of data blocks varies significantly at different stages. For example, during the global model distribution phase, model parameter data blocks are read concurrently by multiple GPU nodes; during the local training phase, read and write operations on gradient data and sample data are primarily concentrated on each node; and during the model aggregation phase, updated gradient data needs to be written and read centrally. However, current technologies for managing storage resources in GPU cluster environments often employ static configuration or simple caching strategies. For example, pre-allocating a fixed amount of local storage resources to each computing node or storing critical data uniformly in a single high-speed storage device. This approach fails to adapt to the dynamic changes in data access frequency during federated learning training and is ill-suited to the frequent switching of access modes between training phases. When data access hotspots are concentrated on specific data blocks, if that data is still located in a low-speed storage layer, it will increase the data access latency of GPU nodes and reduce training efficiency. Conversely, when the hotspot shifts, if the data blocks that previously occupied high-speed storage resources are not migrated back in time, it will lead to capacity shortages in the high-speed storage layer, affecting the loading of new high-frequency data. At the same time, due to the lack of comprehensive awareness of the load status of each storage layer, such as capacity utilization, I / O bandwidth utilization, and access latency, some storage layers may operate under high load continuously, while other storage layers are underutilized, resulting in an overall imbalance in resource allocation. Summary of the Invention

[0004] The purpose of this invention is to provide a method for balancing storage resources in a distributed GPU cluster based on federated learning, thereby addressing the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for balancing storage resources in a distributed GPU cluster based on federated learning, comprising: S1, during the execution of a federated learning training round, collecting read and write operation records of model parameter data blocks, gradient data blocks, and training sample data blocks from each GPU computing node and the central aggregation server, generating real-time access logs, and calculating the access frequency distribution of each data block based on the current training round; S2, comparing the access frequency distribution with a preset heat threshold to identify a set of high-frequency accessed data blocks and forming a hot data list; S3, obtaining real-time load status information of the local high-speed storage layer, the fast cache storage layer, and the shared storage layer, including... S4. Based on the hot data list and the real-time load status of each storage layer, execute the dynamic migration decision algorithm to determine the data blocks to be migrated and their corresponding target storage layers, and generate a migration path plan; S5. When the remaining capacity of the target fast storage layer meets the preset conditions, migrate the high-frequency access data blocks from the low-speed storage layer to the fast storage layer, and update the distributed storage metadata index to establish a new data location mapping relationship; S6. After the data migration is completed, continuously monitor the subsequent access frequency changes of the data blocks. When the access frequency of the data blocks drops below the cold data threshold, migrate them in reverse to the low-speed storage layer to release fast storage resources.

[0006] Preferably, step S1 includes collecting read and write operation records during the execution of federated learning training rounds, extracting access features of model parameters, gradient data, and training samples to obtain an initial access log; identifying data block identifiers based on the initial access log and generating a structured access log; statistically analyzing the read and write counts of each data block through the structured access log to determine the access frequency of a single block; if the access frequency of a single block is greater than a preset threshold, it is marked as a high-frequency data block and a block activity set is obtained; the proportion of high-frequency data blocks is aggregated based on the block activity set, and the access frequency distribution of each data block is calculated.

[0007] Preferably, step S2 includes obtaining a frequency distribution matrix constructed based on read / write weights within a preset time window, the frequency distribution matrix being derived from the extraction of access frequency distribution; processing the frequency distribution matrix using principal component analysis to extract data block feature vectors, and calculating the single-block activity value based on the data block feature vectors; if the single-block activity value is greater than a preset heat threshold, then marking high-frequency access data blocks to obtain a high-frequency access data block set; extracting node identifiers based on the high-frequency access data block set and performing a list construction operation on the aggregation node to generate a hot data list.

[0008] Preferably, step S3 includes obtaining the input / output request queue depth and data throughput per unit time of the storage node; constructing a traffic pressure model based on the input / output request queue depth and data throughput per unit time to obtain the average access latency index and instantaneous input / output bandwidth utilization; obtaining the storage layer's capacity occupancy rate; inputting the capacity occupancy rate, average access latency index, and instantaneous input / output bandwidth utilization rate into a multi-dimensional state evaluation matrix to obtain a resource contention degree value; judging the load status based on the resource contention degree value; and generating real-time load status information including capacity occupancy rate, instantaneous input / output bandwidth utilization rate, and average access latency index.

[0009] Preferably, step S4 includes obtaining access frequency and remaining capacity, and processing the access frequency and remaining capacity to obtain an activity score; if the activity score is greater than a preset threshold, then obtaining the data block to be migrated; obtaining the size and write speed of the data block to be migrated to determine the target storage layer; obtaining the network bandwidth of the target storage layer and the read latency of the data block to be migrated, and executing a dynamic migration decision algorithm based on the network bandwidth and the read latency to generate a migration path plan.

[0010] Preferably, step S5 includes obtaining a heat feature vector based on the read / write frequency of data blocks in the low-speed storage layer; if the heat feature vector is greater than a preset threshold, then determining a high-frequency access data block; obtaining a target fast storage layer based on the volume of the high-frequency access data block and the available space of the fast storage layer; generating a new data location mapping relationship based on the physical address of the target fast storage layer and the hash information of the high-frequency access data block; updating the distributed storage metadata index based on the new data location mapping relationship, thereby migrating the high-frequency access data block from the low-speed storage layer to the fast storage layer and establishing a new data location mapping relationship.

[0011] Preferably, step S6 includes obtaining a popularity score for the first data block based on the number of read / write requests for the first data block; if the popularity score is less than a preset threshold, the first data block is determined to be a data block to be downgraded.

[0012] Preferably, step S6 further includes obtaining the target low-speed node based on the volume of the data block to be downgraded and generating a migration instruction; writing the data block to be downgraded to the target low-speed node according to the migration instruction, updating the index, completing the reverse migration of the data block to the low-speed storage layer and releasing the fast storage resources.

[0013] Preferably, it also includes S7, which involves synchronously updating the cache pointers and data access paths of each GPU computing node according to the migration results, completing the reallocation of storage resources, and making the load of each storage layer tend to be balanced. Specifically, this includes obtaining the data migration completion log, parsing the data migration completion log to extract the unique identifier of the data block and the target physical storage address; retrieving the list of GPU computing nodes holding the old version of the cache pointer according to the unique identifier of the data block; and sending pointer update instructions to the nodes in the list of GPU computing nodes. The pointer update instructions are used to replace the old version of the cache pointer with the new version of the cache pointer pointing to the target physical storage address.

[0014] Preferably, step S7 further includes generating the shortest data access path based on the new cache pointer; collecting traffic data transmitted via the shortest data access path and calculating the storage bandwidth utilization of each storage layer; if the storage bandwidth utilization variance is less than a preset balance threshold, then confirming that the load of each storage layer is in a balanced state.

[0015] As can be seen from the above technical solution, the present invention has the following beneficial effects: This distributed GPU cluster storage resource balancing method based on federated learning, in a federated learning distributed GPU cluster training environment, dynamically identifies high-frequency data and performs cross-storage layer migration by real-time statistical analysis of the access frequency of model parameter data blocks, gradient data blocks, and training sample data blocks within the training rounds. Combined with load status information such as capacity utilization, I / O bandwidth utilization, and access latency of the multi-layer storage architecture, it performs reverse migration when access frequency decreases. This enables adaptive data flow between the local high-speed storage layer, the fast cache storage layer, and the shared storage layer, effectively avoiding GPU computation waiting caused by high-frequency data remaining in the low-speed storage layer, reducing storage resource congestion caused by hotspot switching, and improving the utilization efficiency of high-speed storage resources. Simultaneously, by updating the distributed storage metadata index and synchronously refreshing the cache pointer, it ensures the consistency and stability of data access paths across GPU computing nodes, ultimately achieving dynamic balanced allocation of storage resources, reducing the overall training time of federated learning rounds, and improving training performance and system stability in the distributed GPU cluster environment. Attached Figure Description

[0016] Figure 1 This is a flowchart of the distributed GPU cluster storage resource balancing method of the present invention; Figure 2 This is a diagram illustrating the dynamic migration process of thermal data in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: As Figure 1 and Figure 2 As shown, this invention provides a technical solution: a method for balancing storage resources in a distributed GPU cluster based on federated learning, comprising: S1. During the execution of federated learning training rounds, collect the read and write operation records of each GPU computing node and the central aggregation server on the model parameter data block, gradient data block and training sample data block, generate real-time access logs, and calculate the access frequency distribution of each data block based on the current training round. S2. Based on the comparison between the access frequency distribution and the preset heat threshold, identify the set of high-frequency access data blocks and form a hot data list; S3. Obtain real-time load status information of the local high-speed storage layer, fast cache storage layer and shared storage layer, including capacity utilization, I / O bandwidth utilization and access latency indicators; S4. Based on the hot data list and the real-time load status of each storage layer, execute the dynamic migration decision algorithm to determine the data blocks to be migrated and their corresponding target storage layers, and generate a migration path plan. S5. When the remaining capacity of the target fast storage layer meets the preset conditions, the high-frequency access data blocks are migrated from the low-speed storage layer to the fast storage layer, and the distributed storage metadata index is updated to establish a new data location mapping relationship. S6. After the data migration is completed, continuously monitor the changes in the subsequent access frequency of the data blocks. When the access frequency of the data blocks drops below the cold data threshold, migrate them in reverse to the low-speed storage layer to release fast storage resources. S7. Based on the migration results, the cache pointers and data access paths of each GPU computing node are updated synchronously to complete the reallocation of storage resources and make the load of each storage layer tend to be balanced.

[0019] In S1, an access monitoring module is embedded in the storage I / O path of the GPU computing node to intercept and statistically analyze read and write system calls, generating real-time access logs containing timestamps, data block identifiers, access types, and the number of bytes accessed. Using the training round as a time window, the number of accesses to each data block within the current round is statistically analyzed, forming an access frequency distribution vector. Access frequency can be calculated using the number of accesses per unit time or a weighted number of accesses, with the weights set based on the differences in read and write types.

[0020] In S2, the statistically obtained access frequency is compared with a preset popularity threshold. The popularity threshold can be adaptively set based on historical statistical data or system performance models; for example, the top K percentiles of the access frequency distribution can be taken as the set of high-frequency access data blocks. This forms a hot data list, which is then prioritized.

[0021] In S3, the storage monitoring module collects real-time operational status information for each storage layer. The local high-speed storage layer can be GPU memory or a local NVMe solid-state drive, the fast cache storage layer can be a node-level SSD array or a distributed caching system, and the shared storage layer can be a distributed file system or an object storage system. Capacity utilization, I / O bandwidth utilization, and average access latency metrics are obtained for each layer, forming a multi-dimensional load status vector.

[0022] In S4, a dynamic migration decision-making algorithm model is constructed. This algorithm comprehensively considers data block access frequency, data block size, remaining capacity of the target storage layer, and bandwidth load, and calculates the migration benefit function through a multi-objective optimization strategy. Migration path planning determines the optimal transmission path based on the cluster network topology, prioritizing links with lower bandwidth utilization for data migration.

[0023] In S5, a data migration operation is triggered when the remaining capacity of the target fast storage layer exceeds the sum of the total size of the data blocks to be migrated and the safety redundancy space. The migration process employs a block replication and consistency verification mechanism. After the migration is completed, the distributed storage metadata index is updated, including the mapping relationship between data block identifiers and physical storage locations, and synchronized to each GPU compute node through a consistency broadcast mechanism.

[0024] In S6, a sliding time window is used to continuously monitor changes in the access frequency of data blocks. When the access frequency of a data block falls below a preset cold data threshold, it is migrated from the fast storage layer back to the low-speed shared storage layer to release high-speed storage resources and achieve dynamic reclamation of storage resources.

[0025] In S7, based on the updated data location mapping relationship, the cache pointers and data access path configurations of each GPU computing node are updated synchronously, so that subsequent accesses directly point to the latest data location, avoiding invalid addressing, thereby completing the dynamic balanced scheduling of storage resources of the entire distributed GPU cluster.

[0026] Example 2: S1 includes collecting read and write operation records during the execution of federated learning training rounds, extracting access features of model parameters, gradient data, and training samples to obtain initial access logs; identifying data block identifiers based on the initial access logs to generate structured access logs; statistically analyzing the read and write counts of each data block through the structured access logs to determine the access frequency of a single block; if the access frequency of a single block is greater than a preset threshold, it is marked as a high-frequency data block and a block activity set is obtained; the proportion of high-frequency data blocks is aggregated based on the block activity set to calculate the access frequency distribution of each data block.

[0027] In this embodiment, during each training round of federated learning, each graphics processor computing node and the central aggregation server synchronously form read and write operation records for the read and write operations of the model parameter data block, gradient data block and training sample data block. The read and write operation records are generated at the moment an access action occurs, and the record content includes at least the time of access, the identifier of the computing node or the identifier of the central aggregation server that initiated the access, the type of the accessed data object, the logical location range of the accessed data object in storage, whether the access direction is a read operation or a write operation, and the length of the accessed bytes.

[0028] Specifically, the access time is determined using a monotonically increasing clock source and stored at millisecond granularity to ensure the traceability of access order within the same training round. The compute node identifier and the central aggregation server identifier use unique cluster IDs bound at task startup to prevent confusion during subsequent aggregation by source. The logical location range is determined by the access start offset and the access length, with the access length directly taken from the length field of the read / write request to avoid errors introduced by secondary conversion. The read / write operation records are merged into the initial access log in chronological order. The initial access log is segmented by training rounds, with the start and end boundaries of each training round defined by the round switching signal during training, thus ensuring that the initial access log naturally corresponds to the current training round.

[0029] Furthermore, after obtaining the initial access log, the data block identifiers are identified for each entry in the initial access log, and a structured access log is generated.

[0030] Specifically, the data block identifier is determined by the data object category and the data block number. The data object category is one of the following: model parameters, gradient data, or training samples. The data block number is calculated by the logical location range and the block size. The block size is preset before the training task begins. The determination of the block size follows two constraints: the first constraint is that the block size is not less than the minimum input / output granularity of the underlying storage system to ensure that a single block access does not trigger additional alignment splits; the second constraint is that the block size is not greater than the carrying capacity limit of a single transmission of the cluster network to ensure that single block migration and single block access transmission do not trigger slice reassembly.

[0031] The values ​​of the above two constraints are derived from the storage and network parameter configurations during the deployment phase. The block size is the minimum value that satisfies both constraints, thus balancing storage alignment and network transmission efficiency. For each initial access log record, the starting block number it falls into is first calculated based on the access start offset, and then the ending block number it falls into is calculated based on the access end offset. When the starting block number and the ending block number are the same, a structured access log record is directly generated and assigned a corresponding data block identifier. When the two are different, the access record is expanded into multiple structured access log records according to the block boundaries. Each record corresponds to a data block identifier that is covered, and the byte length of this access is allocated according to the actual byte range covered by each block. The allocation rule is calculated based on the proportion of the byte length covered by the block to the total byte length of the access.

[0032] Furthermore, after the structured access logs are generated, the read and write counts for each data block are summarized and statistically analyzed based on the data block identifiers to determine the access frequency of a single block. The summary statistics use the current training round as the sole statistical window, and accumulate the number of read operations and write operations for all structured access log records appearing for the same data block identifier. The accumulation process is performed by counting the number of records to ensure that the statistical results are not affected by fluctuations in the byte length of a single access.

[0033] Specifically, the single-block access frequency is calculated after the cumulative number of read and write operations is completed. The calculation process adopts a clear weighted synthesis method: the number of write operations is multiplied by a preset weight and then added to the number of read operations to obtain the single-block access frequency. The preset weight is determined to be 2 in the preset stage to reflect the higher impact of write operations on storage bandwidth usage and access latency compared to read operations. The weight value is obtained based on the read and write benchmark measurement of the target storage system during the deployment stage. The benchmark measurement is performed under the same data block size and the same concurrency conditions, and the difference in the impact of the two on bandwidth usage and average access latency is compared. When the impact of write operations is consistently higher than that of read operations, the weight is set to 2, making the single-block access frequency more sensitive to write operations, thereby better reflecting the actual load pressure.

[0034] Furthermore, after obtaining the single-block access frequency of each data block, the single-block access frequency is compared with a preset threshold to mark high-frequency data blocks and form a block activity set.

[0035] Specifically, the preset threshold is generated in each training round according to the following determination process: First, the access frequency of each data block in the current training round is sorted in descending order; then, the upper limit of the number of high-frequency data blocks is determined. The upper limit of the number is calculated from the upper limit of the amount of data that the fast storage layer can use to carry hot data. The upper limit of the amount of data is taken as the value after deducting the reserved capacity from the remaining capacity of the fast storage layer. The reserved capacity is set to 1 / 10 of the total capacity of the fast storage layer to reserve necessary operating margin; then, the data blocks with the highest access frequency are sequentially included in the candidate set, and the amount of data corresponding to the candidate set is accumulated simultaneously. When the accumulated amount of data reaches the upper limit of the amount of data, the inclusion stops; the access frequency of the data block corresponding to the stopping position is determined as the preset threshold in this training round.

[0036] After determining the preset threshold using the above method, if the access frequency of any single data block is strictly greater than the preset threshold, the data block is marked as a high-frequency data block and included in the block activity set. The block activity set is maintained with data block identifiers as elements. The maintenance process only includes inclusion and deduplication to ensure that each high-frequency data block identifier in the set is unique.

[0037] Furthermore, after the block activity set is formed, the proportion of high-frequency data blocks is aggregated based on the block activity set, and the access frequency distribution of each data block is calculated. The calculation process for the proportion of high-frequency data blocks first counts the number of high-frequency data blocks in the block activity set, then counts the total number of unique data blocks that have been accessed in the current training round, and then divides the number of high-frequency data blocks by the total number of unique data blocks to obtain the proportion of high-frequency data blocks. The proportion is rounded to 6 decimal places and processed according to the rounding rules to ensure consistent accuracy in comparisons between rounds.

[0038] Specifically, the calculation process of the access frequency distribution first sums the access frequencies of all data blocks in the current training round to obtain the total access frequency. Then, for each data block, its access frequency is divided by the total access frequency to obtain the access ratio of that data block in the current training round. The access ratio is also rounded to 6 decimal places according to the rounding rules. This forms the access frequency distribution result with the data block identifier as the index and the access ratio as the value. The access frequency distribution result is compared with the high-frequency data blocks.

[0039] Example 3: S2 includes obtaining a frequency distribution matrix constructed based on read and write weights within a preset time window, the frequency distribution matrix being derived from the extraction of access frequency distribution; using principal component analysis to process the frequency distribution matrix to extract data block feature vectors, and calculating the single block activity value based on the data block feature vectors; if the single block activity value is greater than a preset heat threshold, then high-frequency access data blocks are marked to obtain a high-frequency access data block set; extracting node identifiers based on the high-frequency access data block set and performing a list construction operation on the aggregation node to generate a hot data list.

[0040] In this embodiment, when constructing the frequency distribution matrix within a preset time window, the access frequency distribution is first divided into time segments so that each segment of the access frequency distribution covers only one time window.

[0041] Specifically, the length of the preset time window is determined before the training task starts. It is determined based on the minimum complete coverage period of the three types of access behaviors in time within the federated learning training round: distribution of model parameter data blocks, reading of training sample data blocks, and return of gradient data blocks. Specifically, in the first 3 training rounds after the training task starts, the duration from the start to the end of each round is recorded. The maximum value of these 3 durations is taken as the time window length, so that each time window covers the peak and fall process of access within a complete round, avoiding the dilution of activity calculation caused by splitting the concentrated access of the same round into adjacent windows.

[0042] Once the time window length is determined, the aggregation node segments the continuous time axis according to the length and assigns a continuously increasing time window number to each segment.

[0043] As a preferred embodiment, the frequency distribution matrix is ​​derived from the extraction of the access frequency distribution. The extraction process uses the data block identifier as the smallest statistical unit. Within each time window, the number of read operations and write operations for each data block are counted separately, and the read and write counts are weighted and synthesized within the time window to form a uniform access intensity within the window.

[0044] Specifically, the read and write weights are determined before the training task starts, with the read operation weight set to 1 and the write operation weight set to 2. The write operation weight of 2 is determined by performing benchmark measurements on read and write operations with the same data block size and the same concurrent access conditions during the deployment phase, and recording the average access latency and bandwidth utilization under stable conditions. When the increase in average access latency and bandwidth usage of write operations relative to read operations is consistently higher than 1.5 times but lower than 2.5 times, the write operation weight is set to 2. This ensures that the weighted statistics clearly distinguish the differences in resource usage caused by write operations, while avoiding excessive weight that could compress the impact of read operations.

[0045] In addition, after generating a weighted statistical value for each data block within each time window, the weighted statistical values ​​of all data blocks within that time window are summed to obtain the total weighted access volume of the window. Then, for each data block, the weighted statistical value of the data block is calculated as a ratio to the total weighted access volume of the window to obtain the weighted access percentage of the data block within that time window, and this percentage is retained to 6 decimal places to ensure uniform precision.

[0046] After completing the above calculations, using the time window as the row dimension and the data block identifier as the column dimension, the weighted access ratio of each data block within each time window is filled into the corresponding row and column positions to form a frequency distribution matrix; when no access record appears for a data block within a certain time window, 0 is filled into that position.

[0047] When performing principal component analysis on the frequency distribution matrix, the aggregation node first centers the matrix so that subsequent decomposition focuses on the fluctuation characteristics of each data block within different time windows.

[0048] Specifically, the centralization process is as follows: For each data block's corresponding matrix column, first calculate the average value of that column across all time window rows. Then, subtract this average value from the original value of each row in that column, and use the difference as the centralized column value. After centralization, the aggregation node calculates the covariance relationship between columns based on the centralized matrix. The calculation process is as follows: For any two columns, multiply the centralized values ​​of the two columns in the same row, sum the products of all rows, and then average them by subtracting 1 from the row number to obtain the covariance value of the two columns. Arrange the covariance values ​​of all column pairs according to their column indices to form a covariance matrix. After the covariance matrix is ​​formed, the aggregation node decomposes the covariance matrix to obtain a set of principal component directions and their corresponding importance values. The importance values ​​are used to measure the explanatory power of the principal component direction for the original fluctuation information. The aggregation node arranges all principal components from high to low importance values ​​and determines the number of principal components to retain.

[0049] The number of principal components retained is determined by a preset information retention ratio of 0.9. This 0.9 is determined before the training task starts. The determination method is as follows: during the deployment phase, a representative training process is selected, and the information retention ratio is set to 0.8, 0.85, 0.9, and 0.95 for comparison. For each value, the total difference between the frequency distribution matrix after principal component restoration and the original matrix in each row is calculated, and the ratio of the total difference to the total difference in the original matrix is ​​calculated. When the ratio is not higher than 0.05, it is determined that the information retention meets the requirements. The minimum information retention ratio that meets the requirements is taken as the preset information retention ratio, thereby forming a stable balance between dimensionality compression and discriminative retention.

[0050] After determining the number of principal components, the aggregation node arranges the projection values ​​of each data block in the direction of the retained principal components according to the order of the principal components, thus obtaining the feature vector of the data block.

[0051] In this embodiment, the activity value of a single block is calculated based on the feature vector of the data block. The calculation process is completed at the aggregation node and executed one by one for each data block.

[0052] First, to avoid the cancellation of activity levels due to positive and negative projection directions, the absolute value of the projection values ​​of each dimension of the feature vector of the data block is taken. Then, to reflect the differences in the contribution of different principal components to the overall fluctuation information, the absolute value of each dimension is weighted according to the importance ratio of the corresponding principal component. The importance ratio is calculated by taking the ratio of the importance value of the principal component to the sum of the importance values ​​of all retained principal components. Finally, the weighted results of the data block in each dimension are added together to obtain the single-block activity value of the data block.

[0053] The above processing enables the activity value of a single block to simultaneously reflect the contribution of the access fluctuation amplitude and the dominant direction of the fluctuation, thereby forming a comparable activity scale between different data blocks.

[0054] When comparing the activity value of a single block with the preset heat threshold, the preset heat threshold is determined by the aggregation node after each round of matrix processing. The threshold determination process is related to the total activity within the current time window.

[0055] The specific process is as follows: The aggregation node first sorts all data blocks from high to low according to the single block activity value, and calculates the sum of the single block activity values ​​of all data blocks as the total activity value; then, starting from the first sorted position, the single block activity value is accumulated sequentially, and the proportion of the accumulated value to the total activity value is calculated simultaneously. When the proportion first reaches 0.8, the accumulation stops, and the single block activity value of the data block corresponding to the stop position is determined as the preset heat threshold.

[0056] The aforementioned 0.8 is determined before the training task begins. The determination method is as follows: During the deployment phase, representative training processes are selected, and comparisons are made using 0.7, 0.75, 0.8, and 0.85 respectively. For each value, the weighted access percentage of the high-frequency access data block set selected by that value is calculated in the subsequent training round, and the set size is also calculated. When the coverage percentage is not less than 0.8 and the set size fluctuates by no more than 0.05 in three consecutive comparisons, the value is deemed to meet the coverage and stability requirements, and the minimum value meeting the requirements is taken as 0.8. After the threshold is determined, the aggregation node performs a comparison on each data block. Data blocks with a single block activity value greater than the preset popularity threshold are marked as high-frequency access data blocks, and these are aggregated to form a high-frequency access data block set. The set is deduplicated using data block identifiers to ensure that only one marking result is retained for the same data block.

[0057] The hot data list is generated at the aggregation node. The generation process takes a set of frequently accessed data blocks as input and binds them to the node identifier.

[0058] Specifically, for each high-frequency access data block in the set, the aggregation node searches for the access record source field that matches the data block identifier within the entire time window covered by the frequency distribution matrix. It then extracts the graphics processor computing node identifier that initiated the access and the central aggregation server identifier, and performs deduplication on the extracted identifiers to form a set of node identifiers corresponding to the data block. Subsequently, the data block identifier, the set of node identifiers, and the single block activity value are combined into a hot data entry, and all hot data entries are sorted from high to low according to the single block activity value to form a hot data list.

[0059] Example 4: S3 includes obtaining the input / output request queue depth and data throughput per unit time of the storage node; constructing a traffic pressure model based on the input / output request queue depth and data throughput per unit time to obtain the average access latency index and instantaneous input / output bandwidth utilization; obtaining the storage layer capacity utilization rate; inputting the capacity utilization rate, average access latency index, and instantaneous input / output bandwidth utilization rate into a multi-dimensional state evaluation matrix to obtain the resource contention degree value; judging the load status based on the resource contention degree value, and generating real-time load status information including capacity utilization rate, instantaneous input / output bandwidth utilization rate, and average access latency index.

[0060] In this embodiment, when obtaining the input / output request queue depth of a storage node, a sampling interval per unit time is established for each storage node, and a queue depth value is formed within each sampling interval. The unit time value is determined during the training task startup phase. The determination process is as follows: after the training task enters the first training round, the shortest interval between two consecutive bursts of input / output requests is continuously observed. The shortest interval is determined by the moment when the storage node queue depth jumps from a low value to a high value. The shortest interval is divided by 5 and rounded up to the nearest integer second to obtain the unit time value. When the rounding result is less than 1 second, the unit time is 1 second, thereby ensuring that the sampling interval is sufficient to characterize burst changes and that no additional access overhead is caused by overly dense sampling.

[0061] In this embodiment, the queue depth is obtained once at the start and once at the end of each unit of time. The obtained content is the number of input and output requests currently queued by the storage node. The queue depth at the start time and the queue depth at the end time are added together and then divided by 2 to obtain the queue depth value for that unit of time.

[0062] When obtaining the data throughput value per unit time, the statistical caliber uses the number of data bytes transmitted within the same unit time and maintains the same timestamp as the queue depth value.

[0063] Specifically, the throughput value is obtained as follows: at the beginning of the unit time, the cumulative number of read bytes and the cumulative number of write bytes of the storage node are obtained; at the end of the unit time, the cumulative number of read bytes and the cumulative number of write bytes are obtained again; the cumulative number of read bytes at the end time is subtracted from the cumulative number of read bytes at the beginning time to obtain the read throughput bytes within the unit time; the cumulative number of write bytes at the end time is subtracted from the cumulative number of write bytes at the beginning time to obtain the write throughput bytes within the unit time; the read throughput bytes and the write throughput bytes are added together to obtain the total throughput value within the unit time.

[0064] To ensure that the throughput is consistent with subsequent stress modeling, the number of completed requests is obtained simultaneously within the current unit of time. The number of completed requests is the difference between the cumulative number of completed requests within the same time period, and the average single request byte length is obtained based on this. The average single request byte length is obtained by dividing the total throughput value by the number of completed requests. When the number of completed requests is 0, the average single request byte length is 0, and this unit of time is marked as an interval with no effective throughput.

[0065] In this embodiment, when constructing the traffic pressure model based on the input / output request queue depth and the data throughput per unit time, the queue depth and throughput are first unified to the same resource meaning, and then the average access latency index and instantaneous input / output bandwidth utilization are obtained accordingly.

[0066] Specifically, the traffic pressure model uses read and write weights for conversion in the read and write directions. These weights are determined during the system deployment and configuration phase, with a read weight of 1 and a write weight of 2. The determination process is as follows: during deployment and configuration, the rated throughput for both read and write directions provided by the storage nodes is read, and the difference between the rated throughput for write and read directions is calculated. When the rated throughput for write is lower than the rated throughput for read and the difference is within a stable range, the write weight is set to 2 to reflect the higher sensitivity of write direction to bandwidth usage and queuing growth, while the read weight is set to 1 to maintain the read direction measurement benchmark unchanged. The calculation process for instantaneous input / output bandwidth utilization is as follows: first, the upper limit of bandwidth is determined, which is the rated bandwidth published by the storage node. The rated bandwidth is given by the storage node configuration parameters and registered during deployment. Then, the total throughput within a unit of time is divided by the unit time duration to obtain the average bandwidth usage per unit time. This average bandwidth usage is then divided by the upper limit of bandwidth to obtain the instantaneous input / output bandwidth utilization. When the calculation result is greater than 1, it is set to 1, thus ensuring that the bandwidth utilization is within an interpretable range.

[0067] The average access latency metric is obtained by adding queuing wait time to service time. Queuing wait time is determined by queue depth and the number of completed requests. The specific process is as follows: the queue depth value within a unit of time is regarded as the average number of queued requests within that unit of time. Then, the average number of queued requests is divided by the number of completed requests within the unit of time to obtain the average number of times each request waits in the queue. Subsequently, the unit of time is divided by the number of completed requests to obtain the average completion interval time for each request. Finally, the completion interval time is multiplied by the aforementioned number of waiting cycles to obtain the average queuing wait time.

[0068] Service time is used to characterize the time overhead of processing a request on a storage node. Service time is determined by the average single request byte length and the average bandwidth usage per unit time. The specific process is as follows: divide the average single request byte length by the average bandwidth usage per unit time to obtain the average service time; when the average bandwidth usage per unit time is 0, the average service time is 0; add the average queuing time to the average service time to obtain the average access latency index, and attach a timestamp consistent with the unit time to this index.

[0069] Specifically, when obtaining the capacity utilization rate of the storage layer, the occupied capacity value and the total capacity value are obtained by taking the storage layer as the object, and the two are converted into the capacity utilization rate.

[0070] The occupied capacity value is the amount of data that has been allocated and is addressable in the storage layer, and the total capacity value is the upper limit of the available capacity provided by the storage layer. Both values ​​are obtained from the storage layer's capacity statistics at the same time to avoid inconsistencies in occupancy rates caused by reading across different time periods. The occupied capacity value is divided by the total capacity value to obtain the capacity occupancy rate. When the calculated result is greater than 1, it is taken as 1, so that the capacity occupancy rate falls within the range of 0 to 1. When the capacity occupancy rate, average access latency index, and instantaneous input / output bandwidth utilization rate are entered into the multidimensional state evaluation matrix, the multidimensional state evaluation matrix is ​​generated row by row on a unit time basis. Each row corresponds to the three indicators under the same timestamp, the column order is kept consistent, and different storage layers are distinguished by storage layer identifiers.

[0071] As a preferred embodiment, the resource competitiveness value is obtained by normalizing and weighting the multidimensional state assessment matrix.

[0072] Specifically, in the normalization phase, a corresponding baseline threshold is set for each indicator, and the indicator value and the baseline threshold are converted into competitive components in the range of 0 to 1. The baseline threshold for capacity utilization is 0.85. The process for determining the value is as follows: before the training task starts, the total capacity of the storage layer is read, and the total capacity is multiplied by 0.15 to obtain the reserved capacity. Then, the reserved capacity is deducted from the total capacity to obtain the bearable capacity. Finally, the bearable capacity is divided by the total capacity to obtain 0.85. This threshold is used to ensure that the storage layer reserves 0.15 of the capacity for temporary occupation during the data block migration process and space fluctuations during the mapping relationship update.

[0073] The baseline threshold for the average access latency metric is twice the median latency value of historical low-load intervals. The determination process is as follows: In the first 10 units after the training task starts, intervals with instantaneous input / output bandwidth utilization below 0.3 and queue depth not exceeding 1 are selected as low-load intervals. The average access latency metric of these intervals is extracted and sorted by value. The value in the middle of the sort is taken as the median latency value of the low-load interval. Then, the median value is multiplied by 2 to obtain the latency baseline threshold. This threshold is used to identify the state where queuing has increased significantly.

[0074] The baseline threshold for instantaneous input / output bandwidth utilization is set at 0.9. The process for determining this value is as follows: During the deployment and configuration phase, the rated bandwidth of the storage node is used as the upper limit. When the bandwidth utilization reaches 0.9, the throughput per unit time is close to the upper limit and the queue depth is more sensitive to sudden loads. The threshold of 0.9 is used to reflect the intensification of competition in advance before it approaches saturation.

[0075] Furthermore, after setting the baseline threshold, the capacity occupancy rate is divided by 0.85 to obtain the capacity contention component, the average access latency index is divided by the latency baseline threshold to obtain the latency contention component, and the instantaneous input / output bandwidth utilization rate is divided by 0.9 to obtain the bandwidth contention component. Any contention component greater than 1 is taken as 1, thereby avoiding distortion of contention degree due to excessively large single indexes.

[0076] Specifically, in the weighted aggregation phase, weights are assigned to the three contention components: latency contention component with a weight of 0.40, bandwidth contention component with a weight of 0.35, and capacity contention component with a weight of 0.25. The weight determination process is as follows: In the first 20 units of time after the training task starts, the correlation strength between the average access latency index, instantaneous input / output bandwidth utilization, capacity occupancy rate, and the change in training round time is calculated. The correlation strength is measured by the percentage of times the changes are in the same direction. The correlation strengths of the three are sorted from high to low and assigned values ​​of 0.40, 0.35, and 0.25 in high, medium, and low order, respectively. Then, the sum of the three is verified to be 1, ensuring that the resource contention degree value is in the range of 0 to 1. The normalized latency contention component is multiplied by 0.40, the normalized bandwidth contention component is multiplied by 0.35, and the normalized capacity contention component is multiplied by 0.25. The three results are then added together to obtain the resource contention degree value.

[0077] In determining load status based on resource contention levels, a two-tiered threshold system is used for clear classification, and the classification results are used to generate real-time load status information. The high load threshold is set at 0.70, and the congestion load threshold is set at 0.90. These two thresholds are determined within the first 20 units of time after the training task starts. The determination process is as follows: first, the average training round duration over the first 10 units is calculated as a reference duration; then, starting from the 11th unit, intervals are compared one by one. When the training round duration corresponding to a certain unit reaches 1.10 times the reference duration and this phenomenon occurs consecutively for 3 units, the minimum resource contention level value within these 3 units is taken as 0.70; when the training round duration corresponding to a certain unit reaches 1.30 times the reference duration and this phenomenon occurs consecutively for 3 units, the minimum resource contention level value within these 3 units is taken as 0.90. The load status determination rules are as follows: a resource contention value less than 0.70 is considered normal load, a resource contention value not less than 0.70 and less than 0.90 is considered high load, and a resource contention value not less than 0.90 is considered congested load.

[0078] Real-time load status information is generated once at the end of each unit of time. The generated content includes capacity utilization, instantaneous input and output bandwidth utilization, average access latency, and resource contention level, load status classification results, storage layer identifier and timestamp. The timestamp is taken from the generation time and is consistent with the end time of this unit of time interval.

[0079] Example 5: S4 includes obtaining access frequency and remaining capacity, and processing the access frequency and remaining capacity to obtain an activity score; if the activity score is greater than a preset threshold, then obtaining the data block to be migrated; obtaining the size and write speed of the data block to be migrated to determine the target storage layer; obtaining the network bandwidth of the target storage layer and the read latency of the data block to be migrated, and executing a dynamic migration decision algorithm based on the network bandwidth and the read latency to generate a migration path plan.

[0080] In this implementation, before making a dynamic migration decision, the access frequency and remaining capacity are aligned at the same decision time to avoid scoring deviations caused by time inconsistencies.

[0081] Specifically, access frequency is obtained on a per-data-block basis, with the data range limited to the access statistics results corresponding to the current training round. For each data block, the number of read operations and write operations occurring within that round are counted separately. The number of write operations is weighted by 2 and added to the number of read operations to obtain the access frequency of that data block. The weight 2 is determined during the system deployment and acceptance phase by continuously running read and write operations under the same data block size and concurrency conditions, recording the average access latency and bandwidth usage within a stable interval. When the write operation consistently outperforms the read operation in both metrics and the difference is within a repeatable range, the weight for the write operation is set to 2, making the access frequency more sensitive to the resource consumption caused by the write operation. Remaining capacity is obtained separately for each storage layer. The data is obtained by reading the total capacity and occupied capacity of the storage layer at the same decision time, and subtracting the occupied capacity from the total capacity to obtain the remaining capacity. The total capacity and occupied capacity are taken from the capacity statistics provided by the storage layer and include the same collection timestamp to ensure that the remaining capacity reflects the actual available space at that moment.

[0082] Furthermore, access frequency and remaining capacity are processed on the same scale before being included in the activity score calculation, so that the values ​​of different data blocks and different storage layers are comparable.

[0083] Specifically, the same-scale processing of access frequency adopts interval mapping: first, find the highest and lowest access frequency values ​​among all data blocks participating in the decision-making process in this round; then, for a single data block, subtract the lowest access frequency value from its access frequency to obtain the difference, and then divide the difference by the difference between the highest and lowest access frequency values ​​to obtain the access intensity normalized value; when the highest and lowest access frequency values ​​of all data blocks are the same, the access intensity normalized value of all data blocks is set to 0 to avoid meaningless scaling.

[0084] The same-scale processing of remaining capacity adopts a capacity ratio: for each candidate target storage layer, the remaining capacity of the storage layer is divided by the total capacity of the storage layer to obtain the space reserve ratio; when the calculation result is greater than 1, it is taken as 1, and when the result is less than 0, it is taken as 0, so that the space reserve ratio falls within the range of 0 to 1.

[0085] As a preferred embodiment, the activity score is obtained by weighting the access intensity normalized value and the space margin ratio. The synthesis process is performed one by one on each data block to obtain the score value of the data block relative to a certain candidate target storage layer.

[0086] Specifically, the weighting coefficients use an access intensity weight of 0.7 and a space margin weight of 0.3. The 0.7 and 0.3 are determined during the training task activation phase. The determination method is as follows: in 20 consecutive rounds after the start of the training task, the percentage of occurrences of changes in training iteration time and changes in access intensity occurring in the same direction are recorded, and the percentage of occurrences of migration failure due to insufficient space is also recorded. When the percentage of occurrences of changes in training iteration time and changes in access intensity occurring in the same direction is significantly higher than the percentage of occurrences of insufficient space, the access intensity weight is set to 0.7 and the space margin weight is set to 0.3, so that the score mainly reflects the migration benefit and retains the space constraint.

[0087] When calculating the score, the normalized access intensity value is first multiplied by 0.7, then the space margin ratio is multiplied by 0.3, and finally the two results are added together to obtain the activity score. The score is then bound to the data block identifier and the candidate target storage layer identifier to form the input sequence for subsequent threshold screening.

[0088] In this embodiment, the data blocks to be migrated are obtained by comparing the activity score with a preset threshold. The preset threshold is redefined in each decision cycle to ensure that the migration scale is consistent with the available space of the fast storage resources. The threshold determination process is executed in the following order: First, the target storage layer is selected as the fast storage layer, and its remaining capacity is obtained at the same decision time; then, the reserved space is determined, which is 0.15 of the total capacity of the fast storage layer. The 0.15 is determined as follows: during the deployment phase, the peak value of temporary space occupation during the migration process is statistically analyzed. When the peak value of temporary space occupation falls within the range of 0.10 to 0.15 of the total capacity for a long period of time, the reserved space is set to 0.15, so that there is still a clear margin when replication and mapping relationship switching occurs during the migration process; then, the upper limit of the migration space that can be carried in this round is obtained by subtracting the reserved space from the remaining capacity of the fast storage layer.

[0089] Furthermore, after determining the space limit, all data blocks are sorted from high to low according to their activity scores, and the data block size is accumulated block by block starting from the first sorted block. The data block size is the actual number of bytes occupied by the data block in storage. When the accumulated data block size first reaches the upper limit of the migration space, the accumulation stops, and the activity score of the data block corresponding to the stop position is set to a preset threshold.

[0090] After setting the threshold, a comparison is performed on each data block. Data blocks with an activity score greater than the preset threshold are identified as data blocks to be migrated, and duplicates are removed according to the data block identifier to form a set to be migrated.

[0091] In addition, the determination of the target storage layer is based on the size of the data block to be migrated and the data fall rate of the target storage layer, and is constrained by the remaining capacity. For each data block to be migrated, the data fall rate of the candidate storage layer at the current moment is obtained. The data fall rate is taken from the write throughput in the unit time data throughput statistics of the storage layer and aligned with the instantaneous load status timestamp of the storage layer to ensure that the speed reflects the real carrying capacity at that moment.

[0092] Then, the estimated time to fall into each candidate storage layer is calculated: the data block size is divided by the data fall rate of the candidate storage layer to obtain the estimated time; when the data fall rate is 0, the estimated time is set to the maximum value so that the candidate storage layer does not enter the selection in this round.

[0093] Specifically, capacity constraints employ a dual verification process: first, it verifies that the remaining capacity of the candidate storage layer is not less than the data block size; second, it verifies that the remaining capacity of the candidate storage layer is not less than the data block size plus a space buffer. The space buffer is set to 0.10 of the data block size. The 0.10 is determined as follows: during the deployment phase, the additional space increment caused by alignment and metadata usage during migration is statistically analyzed. If the additional space increment does not exceed 0.10 of the data block size for an extended period, the space buffer is set to 0.10, ensuring that the space verification covers this increment. Candidate storage layers that pass the capacity verification are entered into a comparison set. In the comparison set, the storage layer with the shortest expected landing time is selected as the target storage layer for the data block, and a mapping relationship is established between the data block identifier and the target storage layer identifier.

[0094] In this embodiment, the migration path plan is generated after the source storage layer and the target storage layer are determined. The path plan executes a dynamic migration decision algorithm based on the network bandwidth of the target storage layer and the read latency of the data block to be migrated.

[0095] Specifically, the network bandwidth is taken as the available bandwidth between the source location and the target storage layer location. The data is obtained by reading the available bandwidth statistics of the link at the current decision time and distinguishing different transmission segments by the link identifier to ensure that the network state at the same time is used during path evaluation. The read latency is taken as the read latency statistics of the data block to be migrated on the source storage layer. The statistical method is to extract the read completion latency of the data block in the most recent access record, and calculate the arithmetic mean of the 10 values ​​to obtain the read latency. The 10 values ​​are determined as follows: a typical data prefetch and parameter fetch burst in the training round usually contains no less than 10 consecutive reads. Using the average of 10 values ​​can cover a burst and suppress single jitter.

[0096] The dynamic migration decision algorithm evaluates the total migration time for each candidate path. The evaluation process is divided into three stages: the reading stage time is the aforementioned reading latency; the transmission stage time is determined by the data block size and the available bandwidth on the path. Specifically, the transmission time is obtained by dividing the data block size by the effective bandwidth of the path, and the effective bandwidth is the smallest available bandwidth value in the path to avoid overestimating the path capacity; the landing stage time is the aforementioned estimated landing time; the three stages are added together to obtain the total migration time of the candidate path, and the total migration time is bound to the transmission segment sequence traversed by the path to form a candidate result; if multiple candidate paths have the same total migration time, the path with higher effective bandwidth and lower reading latency is selected to shorten the migration traffic time and reduce interference with training access.

[0097] In this embodiment, the output of the migration path planning is generated in the form of an executable path description. The output includes the source storage layer identifier, the target storage layer identifier, the identifier of the data block to be migrated, the transmission segment sequence, the available bandwidth value corresponding to each transmission segment, and the upper limit of bandwidth usage for this migration. The upper limit of bandwidth usage is 0.30 of the available bandwidth of the transmission segment. The method for determining 0.30 is as follows: during the deployment phase, the peak ratio of network bandwidth usage by training communication and data reading is statistically analyzed. When the peak usage is not lower than 0.70 for a long period of time, the upper limit of migration usage is set to 0.30 to ensure the priority of training traffic.

[0098] The above path planning results provide a clear basis for destination selection and transmission constraints for subsequent migration execution, and ensure that the migration decision is based on the same time reference as the access frequency, remaining capacity, network bandwidth, and read latency at that time.

[0099] Example 6: S5 includes obtaining a hot feature vector based on the read / write frequency of data blocks in the low-speed storage layer; if the hot feature vector is greater than a preset threshold, then a high-frequency access data block is determined; the target fast storage layer is obtained based on the volume of the high-frequency access data block and the available space of the fast storage layer; a new data location mapping relationship is generated based on the physical address of the target fast storage layer and the hash information of the high-frequency access data block; the distributed storage metadata index is updated based on the new data location mapping relationship, thereby realizing the migration of high-frequency access data blocks from the low-speed storage layer to the fast storage layer and establishing a new data location mapping relationship.

[0100] In this embodiment, when performing high-frequency access data block identification, the statistical range is first limited to the data blocks registered in the low-speed storage layer, and the read and write frequency formed within the same statistical interval is used as the calculation input.

[0101] Specifically, the statistical interval is taken as the start and end time of the current training round, and the start and end times are defined by the control time of the training round switching, so as to ensure that the same batch of read and write frequencies correspond to the access behavior within the same round.

[0102] Specifically, the number of read operations and write operations are accumulated for each data block, with the accumulation based on the number of read operation records and write operation records of the data block within the statistical interval. When an access covers multiple data blocks, it is first split into multiple records according to the data block boundaries, and then the read operation or write operation count of the corresponding data block is counted separately, so that the read and write frequency of each data block only reflects the access load of the data block itself.

[0103] Furthermore, after obtaining the read and write frequencies of each data block in the low-speed storage layer, a heat feature vector is formed based on the read and write frequencies. The heat feature vector is expressed by an ordered pair of values ​​consisting of read intensity values ​​and write intensity values. The read intensity values ​​are used to characterize the density of read accesses, and the write intensity values ​​are used to characterize the density of write accesses.

[0104] Specifically, the calculation process for the read intensity value is as follows: find the maximum number of read operations within the current statistical interval in all data blocks of the low-speed storage layer; for any data block, divide the number of read operations of that data block by the maximum value to obtain the read intensity value of that data block, and limit the read intensity value to the range of 0 to 1; the calculation process for the write intensity value is as follows: find the maximum number of write operations within the current statistical interval in all data blocks of the low-speed storage layer; for any data block, divide the number of write operations of that data block by the maximum value to obtain the write intensity value of that data block, and limit the write intensity value to the range of 0 to 1.

[0105] The above maximum value is obtained by scanning all data blocks in one round. During the scan, the data block identifier is used to remove duplicates to avoid duplicate counting. After the read intensity and write intensity are calculated, the read intensity value is placed first and the write intensity value is placed last to form the heat feature vector of the data block, so that the heat feature vector corresponds one-to-one with the data block identifier in the numerical dimension.

[0106] Since the judgment condition uses a heat feature vector greater than a preset threshold, the heat feature vector is single-valued during the judgment to obtain a comprehensive heat value for comparison, and this comprehensive heat value is compared with the preset threshold.

[0107] Specifically, the calculation process for the comprehensive popularity score is as follows: the read intensity score is weighted at 0.6, and the write intensity score is weighted at 0.4. The two weighted results are then added together to obtain the comprehensive popularity score. The weights of 0.6 and 0.4 are determined before the training task starts. The determination rule is to normalize the influence coefficients of read access and write access on the training process and then allocate them. The influence coefficient of read access is the proportion of access records generated by the training sample data block and the model parameter data block in one training round, and the influence coefficient of write access is the proportion of access records generated by the gradient data block in the same round. When the influence coefficient of read access is greater than the influence coefficient of write access, the read weight is set to 0.6 and the write weight is set to 0.4, so that the comprehensive popularity score is more in line with the dominant access direction in the round and ensures that the sum of the two weights is 1, so that the comprehensive popularity score falls within the range of 0 to 1.

[0108] In this embodiment, the preset threshold is determined according to the volume that the fast storage layer can support within each statistical interval, so that the total volume of the data blocks that are determined to be frequently accessed matches the fast storage resources; before determining the preset threshold, the available space of the fast storage layer is calculated first, and the upper limit of the volume for this migration is formed.

[0109] The calculation process for available space is as follows: At the same time, obtain the total capacity and occupied capacity of the fast storage layer, and subtract the occupied capacity from the total capacity to obtain the remaining space; then calculate the reserved space and subtract it from the remaining space to obtain the upper limit of the volume that can be carried; the reserved space is obtained by adding two parts: the first part is the concurrent migration overlap occupied space, which is twice the size of the largest data block in the candidate migration set. The multiple of 2 is determined based on the fact that the original copy of the low-speed storage layer remains valid during the formation of a new copy of the same data block in the fast storage layer, and a rollback buffer is retained before switching the mapping relationship; the second part is the address alignment fragment space, which is the minimum allocation granularity of the fast storage layer multiplied by the number of data blocks in the candidate migration set. The minimum allocation granularity is the smallest contiguous allocation unit published by the fast storage layer. The product is used to cover the alignment occupancy generated when allocating each data block.

[0110] Furthermore, after completing the reserved space calculation, the reserved space is subtracted from the remaining space to obtain the upper limit of the carrying volume; when the subtraction result is negative, the upper limit of the carrying volume is set to 0, indicating that the migration to the fast storage layer will not be triggered in this statistical interval.

[0111] Then, all data blocks in the low-speed storage layer are sorted from high to low according to their comprehensive heat index, and the data block volume is accumulated block by block according to the sorting order. The data block volume is the number of bytes occupied by the data block registered in the distributed storage metadata index. When the accumulated volume first reaches the upper limit of the carrying capacity, the accumulation stops, and the comprehensive heat index of the data block corresponding to the stop position is determined as the preset threshold.

[0112] After the preset threshold is determined, a comparison is performed on each data block. Data blocks with a comprehensive heat value greater than the preset threshold are identified as high-frequency access data blocks, thus forming a set of high-frequency access data blocks. The set formation process uses data block identifiers to remove duplicates, ensuring that each data block appears only once in the set.

[0113] Furthermore, after obtaining the set of high-frequency access data blocks, the target fast storage layer is obtained based on the volume of the high-frequency access data blocks and the available space of the fast storage layer. If there are multiple candidate layers for the fast storage layer, the upper limit of the volume that can be carried is calculated for each candidate layer. The calculation method is the same as described above, with the total capacity, occupied capacity and minimum allocation granularity at the same time as input, so as to ensure that the comparison benchmark is consistent.

[0114] Subsequently, the total volume of the high-frequency access data block set is calculated. The total volume is the sum of the volumes of each data block in the set, and the volume is taken from the value registered in the distributed storage metadata index. When the upper limit of the capacity of a candidate fast storage layer is less than the total volume of the set, the candidate layer is not included in the target selection range. When there are multiple candidate layers that meet the condition that the upper limit of the capacity is not less than the total volume of the set, the candidate layer with the largest upper limit of the capacity is selected as the target fast storage layer, so that the continuity constraint of subsequent physical address allocation is easier to meet and fragmentation is reduced.

[0115] Furthermore, after the target fast storage layer is determined, a new data location mapping relationship is generated based on the physical address of the target fast storage layer and the hash information of the high-frequency access data blocks.

[0116] Specifically, the process of obtaining physical addresses is as follows: the high-frequency access data block set is allocated address block by block in sorted order, and the allocation unit is a continuous physical address range; the length of the continuous physical address range of each data block is taken as the volume of the data block, and the starting physical address is selected from the current free address range of the target fast storage layer in order to meet the length requirement, and the range is deducted from the free range set so that subsequent allocations will not repeatedly occupy the same address range.

[0117] The hash information generation process is as follows: For each frequently accessed data block, the full content of the data block is read from the low-speed storage layer according to the data block volume, and a digest value is generated according to a preset digest rule. The digest rule adopts a deterministic digest generation method with a length of 256 bits. The deterministic requirement ensures that the same content will obtain the same digest value at different times, and the length of 256 bits is used to reduce the probability of different content generating the same digest value. The digest generation process divides the data block content into segments according to the smallest allocation granularity, and each segment participates in the digest update in turn until it covers the full content of the data block, thus obtaining the hash information of the data block. The new data location mapping relationship is generated in the form of entries. Each entry contains at least the data block identifier, the target fast storage layer identifier, the continuous physical address range, the hash information, and the generation time. The generation time is the system time when the entry is formed.

[0118] When updating the distributed storage metadata index based on the new data location mapping relationship, the update process is executed one by one on a data block basis and remains traceable.

[0119] First, the original location entry is located in the distributed storage metadata index by the data block identifier, and the original location entry points to the low-speed storage layer location. Then, the storage layer identifier in the original location entry is replaced with the target fast storage layer identifier, and the location field is replaced with a new continuous physical address range. At the same time, the hash information is registered as a verification field, so that subsequent accesses have a consistent verification basis after locating the physical address of the fast storage layer.

[0120] After the entry update is completed, a consistency check is performed: the data block content is read from the target fast storage layer according to the new continuous physical address range, the digest value is regenerated according to the same 256-bit digest rule, and the digest value is compared bit by bit with the hash information registered in the metadata index; if the comparison is consistent, the new data location mapping relationship is confirmed to be effective, and the access location entry of the data block is pointed to the fast storage layer; if the comparison is inconsistent, the entry update is revoked, the metadata index is restored to the original location entry pointing to the low-speed storage layer, and the data block is removed from the set of high-frequency access data blocks in this operation.

[0121] Through the above index updates and consistency checks, high-frequency access data blocks are migrated from the low-speed storage layer to the fast storage layer, and a new data location mapping relationship is established in the distributed storage metadata index.

[0122] Example 7: S6 includes obtaining a heat score for the first data block based on the number of read and write requests for the first data block; if the heat score is less than a preset threshold, the first data block is determined to be a data block to be downgraded; a target low-speed node is obtained based on the volume of the data block to be downgraded and a migration instruction is generated; the data block to be downgraded is written to the target low-speed node according to the migration instruction, the index is updated, the data block is reverse-migrated to the low-speed storage layer and the fast storage resources are released.

[0123] In this embodiment, before determining the popularity score of the first data block, the identity and statistical range of the first data block are locked to avoid the statistical object drifting. The first data block is limited to a data block that is already in the fast storage layer and has a location entry in the index. The index entry at least contains the identifier of the data block and its address in the fast storage layer.

[0124] The statistical interval is aligned with the training rounds, with the statistical start point taken as the beginning time of the training round and the statistical end point taken as the end time of the training round. This ensures that the number of read and write requests only reflects the access behavior within that round. The process of obtaining the number of read and write requests involves filtering access records one by one according to the data block identifier: traversing the read and write operation records within the statistical interval, and counting records where the data block identifier matches the first data block identifier. When the access direction of a record is a read operation, the number of read requests in the number of read and write requests is incremented by 1; when the access direction of a record is a write operation, the number of write requests in the number of read and write requests is incremented by 1. When the same access covers multiple data blocks, it is first split into multiple records according to the data block boundaries, and then the request counts for the corresponding data blocks are counted separately, so that the number of read and write requests for the first data block does not include the access contribution of other data blocks.

[0125] In this embodiment, the popularity score of the first data block is obtained by weighting and normalizing the number of read and write requests. The calculation process uses a single-value result to ensure that the comparison with the threshold has a unique conclusion.

[0126] First, a weighted synthesis is performed: the number of read requests for the first data block is counted with weight 1, and the number of write requests for the first data block is counted with weight 2. The two counts are then added together to obtain the weighted total number of requests for the first data block. Weight 2 is determined during the deployment and acceptance phase. The determination method is to run read load and write load separately under the same data block size and the same number of concurrent requests, and to count the average access latency and data throughput per unit time. When the average access latency of the write direction is greater than the average access latency of the read direction and the data throughput per unit time of the write direction is lower than the data throughput per unit time of the read direction, and the above relationship remains consistent in three consecutive acceptance samplings, the weight of the write direction is set to 2, so that the heat score is more sensitive to changes in the write load. After weighted synthesis is completed, normalization is performed: within the same statistical interval, the total number of weighted requests for each data block in the fast storage layer is calculated according to the same weighting rule, and the maximum total number of weighted requests is found; the heat score of the first data block is obtained by dividing the total number of weighted requests for the first data block by the maximum total number of weighted requests, and the heat score is retained to 6 decimal places. The normalization process makes the heat score fall within the range of 0 to 1 and supports direct comparison between different data blocks; when the maximum total number of weighted requests is 0, the heat score is 0, indicating that no valid access requests have occurred in the fast storage layer within the statistical interval.

[0127] In this embodiment, a preset threshold is used to determine whether the first data block enters the set to be downgraded. The threshold is determined based on the need for rapid storage resource release within each statistical interval, so that the reverse migration is consistent with the space reclamation goal.

[0128] Specifically, before determining the threshold, the amount of space that the fast storage layer needs to release is calculated: at the same decision moment, the total capacity and the occupied capacity of the fast storage layer are obtained, and the current remaining space is obtained by subtracting the occupied capacity from the total capacity; then, the target value for retaining the remaining space is determined, which is set to 0.15 of the total capacity of the fast storage layer. 0.15 is determined during the deployment phase by statistically analyzing the peak value of space overlap occupancy that occurs during a migration. The peak value of space overlap occupancy is generated by the double occupancy during the period when the content of the new location is stored but the index has not been switched, and the fragmentation occupancy caused by address allocation alignment is superimposed. The maximum value of the peak value in 10 consecutive migration observations is taken and compared with the total capacity of the fast storage layer. When the ratio is not greater than 0.15, the target value for retaining the remaining space is set to 0.15.

[0129] Then, the amount of space to be released is obtained by subtracting the current remaining space from the target value of the remaining space. When the amount of space to be released is less than 0, the amount of space to be released is set to 0, indicating that the release-driven degradation is not triggered in the current statistical interval.

[0130] After calculating the amount of space to be released, all data blocks in the fast storage layer are sorted in ascending order of popularity score, and the data block volume is accumulated block by block in sorting order. The data block volume is the number of bytes occupied registered in the index entry. When the accumulated volume reaches the amount of space to be released, the accumulation stops, and the popularity score of the data block corresponding to the stop position is determined as the preset threshold.

[0131] After the threshold is determined, the following judgment is performed: if the popularity score of the first data block is less than the preset threshold, the first data block is determined to be a data block to be downgraded; if the popularity score of the first data block is not less than the preset threshold, the first data block remains in the fast storage layer and is not included in the set to be downgraded this time.

[0132] Furthermore, after determining the data block to be downgraded, the target low-speed node is obtained based on the volume of the data block to be downgraded and a migration instruction is generated. The acquisition of the target low-speed node is based on volume constraints as a prerequisite to ensure that the target side has the capacity.

[0133] First, the volume of the data block to be downgraded is obtained. The volume is taken from the number of bytes occupied registered in the index entry, and is consistent with the volume caliber used in this threshold calculation. Then, the remaining capacity of each low-speed node in the low-speed storage layer is obtained one by one. The remaining capacity is obtained by subtracting the occupied capacity from the total capacity of the low-speed node. When the remaining capacity of a low-speed node is greater than or equal to the volume of the data block to be downgraded, the low-speed node is added to the candidate set. If there is only one low-speed node in the candidate set, the low-speed node is determined as the target low-speed node. If there are multiple low-speed nodes in the candidate set, they are sorted from largest to smallest according to the result of subtracting the volume of the data block to be downgraded from the remaining capacity of the candidate set. The first node in the sort is selected as the target low-speed node to maximize the remaining space after this storage and reduce the probability of subsequent allocation failure. If there is a tie for the first place in the sort, the node with the smallest node identifier value is selected as the target low-speed node.

[0134] In addition, migration instructions are generated after the target low-speed node is determined. Migration instructions are used to constrain the source location, target location and consistency check conditions of the migration execution.

[0135] Specifically, the migration instruction includes at least the identifier of the data block to be downgraded, the identifier of the source fast storage layer, the source address range, the identifier of the target low-speed node, the target address range, the data block size, hash information, the instruction generation time, and the instruction sequence number. The source address range is taken from the physical address range of the fast storage layer recorded in the index entry. The target address range is allocated before the migration instruction is generated. The allocation process is to select a continuous interval with a length not less than the data block size from the free address interval of the target low-speed node, and prioritize the interval with the smallest starting address that meets the length requirement. This interval is then deducted from the free interval set to avoid duplicate allocation.

[0136] Hash information is used for content consistency verification after migration. It is generated by generating a 256-bit digest value for the full content of the data block to be downgraded. The digest rule is set as a deterministic digest rule when the system is deployed. Determinism requires that the digest value generated for the same content is consistent on any node. When the digest is generated, the data block content is segmented according to the smallest allocation granularity of the target low-speed node. The segments participate in the digest update in the order of segmentation until the entire content is covered, and the final digest value is obtained and registered in the migration instruction.

[0137] When performing reverse migration according to the migration instruction, the source content is read, the network is transmitted, the target location content is stored and the consistency is verified first, and then the index is updated and the storage resources are released quickly to ensure that the access location switch is based on the content that has passed the verification.

[0138] During execution, the data block content to be downgraded is retrieved from the source address range indicated by the migration instruction, and the content is transmitted completely to the target low-speed node according to the data block size. After receiving the content, the target low-speed node stores the content according to the target address range indicated by the migration instruction, and generates a 256-bit digest value for the content within the target address range after storage. The digest value is compared bit by bit with the hash information carried by the migration instruction. If the bit-by-bit comparison matches, the index update step is initiated. The index update locates the index entry according to the data block identifier, replaces the storage layer identifier and address range pointing to the fast storage layer in the index entry with the target low-speed node identifier and target address range, and retains the hash information as the basis for subsequent access consistency verification. After the index update is completed, the original source address range of the data block in the fast storage layer is returned to the set of free address ranges in the fast storage layer, completing the release of fast storage resources.

[0139] When the bitwise comparison is inconsistent, the index is not updated. The index entry remains pointing to the original position of the fast storage layer. At the same time, the target address range already allocated to the target low-speed node is returned to its free address range set to avoid the continued existence of invalid occupation.

[0140] Through the above-mentioned processes of heat score calculation, threshold determination, target low-speed node selection, migration instruction generation, content consistency verification and index update, data blocks are reverse-migrated to the low-speed storage layer and the fast storage resources are released.

[0141] Example 8: S7 includes acquiring the data migration completion log, parsing the data migration completion log to extract the unique identifier of the data block and the target physical storage address; retrieving the list of GPU computing nodes holding the old version cache pointer based on the unique identifier of the data block; sending pointer update instructions to the nodes in the list of GPU computing nodes, the pointer update instructions are used to replace the old version cache pointer with the new version cache pointer pointing to the target physical storage address; generating the shortest data access path based on the new version cache pointer; collecting traffic data transmitted through the shortest data access path and calculating the storage bandwidth utilization of each storage layer; if the storage bandwidth utilization variance is less than the preset balance threshold, then confirming that the load of each storage layer is in a balanced state.

[0142] In this embodiment, before performing cache pointer synchronization update, the data migration completion log is first obtained, and the log entries are used as the processing unit to complete parsing and filtering.

[0143] Specifically, the data migration completion log is generated when the data block migration action ends. The log entry includes at least the migration completion time, the unique identifier of the data block, the target physical storage address, and the migration result identifier. The unique identifier of the data block is assigned when the data block is generated and remains unchanged throughout the training period. It is used to locate the same data block across nodes. The target physical storage address is the address after migration. It is expressed using the physical address range on the storage medium and includes at least the starting address and the length.

[0144] During parsing, the migration result identifier is read first. Only entries whose migration result identifier indicates completion are processed, and the remaining entries are directly removed to avoid triggering pointer switching due to incomplete migration. For entries that are processed, the unique identifier of the data block and the target physical storage address are extracted from the entry, and the two are combined into a location update record. The location update record includes the migration completion time of the entry, which is used to determine the order of subsequent instructions and avoid the old logs from overwriting the new logs.

[0145] When retrieving the list of GPU compute nodes holding the old version of the cache pointer based on the location update record, a full node consistency check is performed using the unique identifier of the data block as the search key. The check process is initiated by the central aggregation server and completed by each GPU compute node.

[0146] Specifically, the central aggregation server sends query messages to all GPU compute nodes. The query message only carries the unique identifier of the data block and the migration completion time. The migration completion time is used by the node to compare the update time of the local cache pointer. After receiving the query message, each GPU compute node first locates the corresponding entry in the local cache pointer according to the unique identifier of the data block. If the entry is not located, the node is not included in the list of GPU compute nodes. If the entry is located, the node reads the address pointed to by the entry and the update time of the entry, and performs two comparisons: First, the address pointed to by the entry is compared with the target physical storage address field by field. The fields must include at least the storage layer identifier and the physical address range. If any field is inconsistent, it is determined to be an old cache pointer. Second, the update time of the entry is compared with the migration completion time. If the update time of the entry is earlier than the migration completion time, it is determined that the entry has not yet reflected the result of this migration.

[0147] When any comparison result is the old cache pointer, the GPU compute node sends back the node identifier and the unique identifier of the data block to the central aggregation server. After receiving the feedback, the central aggregation server generates a list of GPU compute nodes and removes duplicate node identifiers to ensure that the same node appears only once in the current update cycle of the same data block.

[0148] When sending pointer update instructions to nodes in the GPU computing node list, the central aggregation server issues instructions to each node in the list order and provides unique and verifiable update elements in the instructions.

[0149] Specifically, the pointer update instruction includes at least a unique identifier for the data block, the target physical storage address, the new cache pointer version number, the instruction sequence number, and the instruction generation time. The new cache pointer version number is generated using an incrementing integer. The generation rule is to use the millisecond timestamp corresponding to the migration completion time as the version base, and then add the incrementing sequence number of the instructions generated by the central aggregation server for different data blocks within the same millisecond, thereby ensuring that the version number is strictly incrementing and non-repeating globally. The instruction sequence number is also generated using an incrementing integer and is used by the node side to identify duplicate instructions and out-of-order instructions.

[0150] After receiving a pointer update instruction, the GPU compute node locates the old cache pointer entry based on the unique identifier of the data block. If the entry does not exist, a new cache pointer entry is created directly, and its pointed address is set to the target physical storage address. At the same time, the version number of the new cache pointer and the instruction generation time are recorded. If the entry exists, the local version number is compared with the instruction version number. If the local version number is less than the instruction version number, a replacement is performed, replacing the pointed address of the old cache pointer with the target physical storage address, and the version number and update time are updated synchronously. If the local version number is not less than the instruction version number, the replacement is rejected to prevent the new pointer from being rolled back.

[0151] After the replacement is completed, the GPU compute node sends a confirmation message back to the central aggregation server. The confirmation message includes at least the node identifier, the unique identifier of the data block, the version number of the new cache pointer that has taken effect, and the confirmation time. The central aggregation server marks the node as having completed the pointer update based on the confirmation message. Nodes that do not send back confirmation remain in the pending update state and receive instructions again in the next retransmission cycle.

[0152] When generating the shortest data access path based on the new cache pointer, the starting point of the shortest path is the GPU computing node that initiates the access, and the ending point is the storage location corresponding to the target physical storage address pointed to by the new cache pointer.

[0153] Specifically, the shortest path is determined using a unique criterion: First, the minimum number of network hops is used as the primary criterion. The number of network hops is the number of forwarding nodes traversed from the GPU computing node to the target storage location, which is obtained by expanding the existing routing information hop by hop. When multiple candidate paths with the same number of network hops exist, the maximum minimum available bandwidth of the candidate path is used as the secondary criterion. The minimum available bandwidth is the minimum value of the available bandwidth of each transmission segment on the path, ensuring that the path capacity is based on the weakest segment. The available bandwidth is obtained by subtracting the bandwidth occupied by the link in the most recent statistical period from the link's upper limit bandwidth. The link bandwidth upper limit is registered as a rated value during deployment, and the occupied bandwidth is calculated from the number of bytes transmitted by the link in the most recent statistical period.

[0154] When generating paths, GPU compute nodes first enumerate the set of candidate paths to the target storage location. The enumeration process is constrained by the reachability of routes, and no unreachable paths are generated. For each candidate path, the network hop count is calculated and the set of paths with the fewest hop counts is selected. Then, the minimum available bandwidth of each path in this set is calculated and the path with the largest minimum available bandwidth is selected as the shortest data access path.

[0155] Once generated, the shortest data access path is bound to the unique identifier of the data block, and subsequent accesses to the data block will be addressed and transmitted according to this path.

[0156] When collecting traffic data transmitted via the shortest data access path and calculating the storage bandwidth utilization of each storage layer, a unified statistical period is used to complete the hierarchical statistics and utilization conversion.

[0157] The statistical period is set by the central aggregation server. The setting rule is to first obtain the duration of the most recent training round, then divide the duration by 10 to obtain the candidate statistical period, and then round the candidate statistical period up to an integer multiple of 1 second, and apply a lower limit of 1 second and an upper limit of 10 seconds to ensure that the statistical period can reflect the traffic changes after migration without introducing excessive statistical overhead. Traffic data collection uses the statistical period as a window. The cumulative transmission byte count of each storage layer is recorded at the beginning of the window, and the cumulative transmission byte count is recorded again at the end of the window. The difference between the two counts is the effective byte amount transmitted by the storage layer through the shortest data access path within the statistical period. The effective byte amount only counts the transmissions whose destination address falls within the range of the target physical storage address of the storage layer, to avoid cross-layer forwarding byte amount interfering with the layering results.

[0158] For each storage tier, the effective byte count is converted into the throughput value of that period according to the statistical period duration, and then converted into the bandwidth limit of that storage tier to obtain the bandwidth utilization rate. The bandwidth limit of the storage tier is determined during the deployment and acceptance phase. The determination method is to record the maximum throughput value when the throughput of the storage tier reaches the stable platform during a continuous period when the storage tier is in a low contention state and the queue depth is stable and does not increase, and register the maximum throughput as the bandwidth limit to ensure that the denominator of the bandwidth utilization rate reflects the sustainable carrying capacity rather than the instantaneous peak.

[0159] The bandwidth utilization rate is calculated and retained to 6 decimal places, and is bound to the statistical period timestamp and storage layer identifier to form a utilization rate set aligned with the same period.

[0160] When determining whether the load of each storage layer is balanced, a sample set is formed by the bandwidth utilization of each storage layer within the same statistical period, and the variance is calculated. The variance is used to characterize the dispersion of the sample set. The smaller the dispersion, the closer the utilization of each layer is.

[0161] Specifically, the variance calculation process is performed using purely numerical steps: First, the average value of the sample set is calculated, which is obtained by summing the bandwidth utilization of all storage layers and then dividing by the number of storage layers; then, the difference between the bandwidth utilization of each storage layer and the average value is calculated, and this difference is multiplied by itself to obtain the difference product; the difference products of all storage layers are summed and then divided by the number of storage layers to obtain the bandwidth utilization variance for that statistical period; the preset balance threshold is set during the deployment and acceptance phase, and the setting process is tied to the training throughput stability interval: 30 consecutive statistical periods in which the training throughput does not continuously decrease and there are no congestion alarms in each storage layer are selected, and the bandwidth utilization variance is calculated period by period using the above method, and the largest variance is taken as the preset balance threshold, so that the threshold corresponds to a discrete upper bound of stable throughput and no single-layer overload.

[0162] During real-time determination, the variance of bandwidth utilization in the current statistical period is compared with the preset balance threshold. If the variance is less than the preset balance threshold, it is confirmed that the load of each storage layer is in a balanced state. If the variance is not less than the preset balance threshold, the migration completion log, pointer update and shortest data access path are continuously executed, so that the access path and pointer status converge with the migration result until the variance falls within the threshold.

[0163] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for balancing storage resources in a distributed GPU cluster based on federated learning, characterized in that, include: S1. During the execution of federated learning training rounds, collect the read and write operation records of each GPU computing node and the central aggregation server on the model parameter data block, gradient data block and training sample data block, generate real-time access logs, and calculate the access frequency distribution of each data block based on the current training round. S2. Based on the comparison between the access frequency distribution and the preset heat threshold, identify the set of high-frequency access data blocks and form a hot data list; S3. Obtain real-time load status information of the local high-speed storage layer, fast cache storage layer and shared storage layer, including capacity utilization, I / O bandwidth utilization and access latency indicators; S4. Based on the hot data list and the real-time load status of each storage layer, execute the dynamic migration decision algorithm to determine the data blocks to be migrated and their corresponding target storage layers, and generate a migration path plan. S5. When the remaining capacity of the target fast storage layer meets the preset conditions, the high-frequency access data blocks are migrated from the low-speed storage layer to the fast storage layer, and the distributed storage metadata index is updated to establish a new data location mapping relationship. S6. After the data migration is completed, continuously monitor the changes in the subsequent access frequency of the data blocks. When the access frequency of the data blocks drops below the cold data threshold, migrate them in reverse to the low-speed storage layer to release fast storage resources.

2. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 1, characterized in that: S1 includes: Collect read and write operation records during the execution of each training round of federated learning, extract model parameters, gradient data, and access features of training samples to obtain the initial access log; Based on the initial access log, identify the data block identifiers and generate structured access logs; The frequency of access to a single block is determined by counting the number of reads and writes to each data block using structured access logs. If the access frequency of a single block is greater than a preset threshold, it is marked as a high-frequency data block and a block activity set is obtained; The access frequency distribution of each data block is calculated based on the proportion of high-frequency data blocks aggregated according to the block activity set.

3. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 1, characterized in that: S2 includes: Obtain the frequency distribution matrix constructed based on the read and write weights within a preset time window. The frequency distribution matrix is ​​derived from the extraction of the access frequency distribution. Principal component analysis is used to process the frequency distribution matrix to extract the feature vectors of the data blocks, and the activity value of each block is calculated based on the feature vectors of the data blocks. If the activity value of a single block is greater than the preset popularity threshold, then the high-frequency access data block is marked to obtain a set of high-frequency access data blocks; Extract node identifiers from the set of frequently accessed data blocks and perform list building operations on the aggregation node to generate a hot data list.

4. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 1, characterized in that: S3 includes: Obtain the depth of the input / output request queue and the data throughput per unit time of the storage node; A traffic pressure model is constructed based on the input / output request queue depth and the data throughput per unit time to obtain the average access latency index and instantaneous input / output bandwidth utilization. Obtain the storage layer capacity utilization rate, input the capacity utilization rate, average access latency index and instantaneous input / output bandwidth utilization rate into the multi-dimensional state evaluation matrix to obtain the resource contention degree value; The load status is determined based on the resource contention level, and real-time load status information including capacity utilization, instantaneous input / output bandwidth utilization, and average access latency is generated.

5. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 1, characterized in that: S4 includes: The access frequency and remaining capacity are obtained, and the access frequency and remaining capacity are processed to obtain an activity score; If the activity score is greater than the preset threshold, then the data block to be migrated is obtained; Obtain the size and write speed of the data blocks to be migrated to determine the target storage layer; Obtain the network bandwidth of the target storage layer and the read latency of the data blocks to be migrated. Execute a dynamic migration decision algorithm based on the network bandwidth and the read latency to generate a migration path plan.

6. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 1, characterized in that: S5 includes: The heat feature vector is obtained based on the read and write frequency of low-speed storage layer data blocks. If the heat feature vector is greater than the preset threshold, the high-frequency access data block is identified. The target fast storage layer is determined based on the volume of the high-frequency access data block and the available space of the fast storage layer. A new data location mapping relationship is generated based on the physical address of the target fast storage layer and the hash information of frequently accessed data blocks; The distributed storage metadata index is updated based on the new data location mapping relationship, enabling high-frequency access data blocks to migrate from the low-speed storage layer to the fast storage layer and establish a new data location mapping relationship.

7. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 1, characterized in that: S6 includes: The popularity score of the first data block is obtained based on the number of read and write requests for the first data block; If the popularity score is less than the preset threshold, the first data block is determined to be a data block to be downgraded.

8. The method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 7, characterized in that: S6 further includes: The target low-speed node is obtained based on the volume of the data block to be downgraded, and migration instructions are generated. According to the migration instructions, the data block to be downgraded is written to the target low-speed node, the index is updated, the data block is reverse-migrated to the low-speed storage layer and the fast storage resources are released.

9. A method for balancing storage resources in a distributed GPU cluster based on federated learning according to claim 7, characterized in that, This also includes S7, which synchronizes and updates the cache pointers and data access paths of each GPU compute node based on the migration results, completing the reallocation of storage resources and balancing the load across storage layers. Specifically, this includes: Obtain the data migration completion log, and parse the data migration completion log to extract the unique identifier of the data block and the target physical storage address; Retrieve the list of GPU compute nodes holding the old cache pointer based on the data block's unique identifier; Send pointer update instructions to the nodes in the GPU compute node list. The pointer update instructions are used to replace the old cache pointer with a new cache pointer that points to the target physical memory address.

10. A method for balancing storage resources in a distributed GPU cluster based on federated learning, as described in claim 9, characterized in that: The S7 also includes: Generate the shortest data access path based on the new cache pointer; Collect traffic data transmitted via the shortest data access path and calculate the storage bandwidth utilization of each storage layer; If the variance of storage bandwidth utilization is less than the preset balance threshold, then the load of each storage layer is confirmed to be balanced.

Images