A dynamic configuration-based distributed storage optimization method and device

Abstract

The application discloses a kind of based on dynamic configuration's distributed storage optimization method and device, it is related to data storage technical field, including: real-time acquisition strip's access frequency determines corresponding current strip heat level;With the strip of current configuration version number's strip heat level and associated current strip heat level is not consistent, based on corresponding current strip heat level update current configuration version number and local repairable code strip configuration parameter;Real-time to each strip in incremental data identification determines change data block;Each change data block is determined by traversing cluster respectively priority scheduling cluster;Local recoding is executed to each change data block, and the local parity block associated with each change data block is updated;Each change data block and corresponding local parity block are written to corresponding priority scheduling cluster, and the corresponding strip is updated.Based on the above scheme, the data storage reliability of distributed storage system is improved as a whole.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and particularly relates to a distributed storage optimization method and device based on dynamic configuration. BACKGROUND

[0002] With the explosive growth of data, distributed storage systems are widely used in data centers and cloud platforms. To improve reliability and repair efficiency, local repair codes have become a popular coding method, but they face problems such as high bandwidth consumption and high computational overhead in the process of stripe merging and reconstruction. At the same time, existing coding schemes use a fixed layout strategy, which is difficult to adapt to dynamic load changes. Therefore, it is necessary to improve the data storage reliability of distributed storage systems. SUMMARY

[0003] The present application provides a distributed storage optimization method and device based on dynamic configuration, which is used to improve the data storage reliability of distributed storage systems.

[0004] The first aspect of the present application provides a distributed storage optimization method based on dynamic configuration, applied to a distributed storage system, the method comprising:

[0005] Real-time collection of access frequencies of each stripe in the distributed storage system, and determination of current stripe heat levels of each stripe based on the access frequencies;

[0006] For the stripe whose current stripe heat level is inconsistent with the stripe heat level of the current configuration version number, updating the current configuration version number and the local repairable code stripe configuration parameters based on the corresponding current stripe heat level;

[0007] Real-time incremental data identification in each stripe to determine the changed data block;

[0008] Traversing the cluster of the distributed storage system to determine the priority scheduling cluster of each changed data block;

[0009] Performing local recoding on each changed data block to update the local parity block associated with each changed data block;

[0010] Writing each changed data block and the corresponding local parity block to the corresponding priority scheduling cluster, and updating the corresponding stripe.

[0011] Optionally, the determination process of the access frequency comprises:

[0012] Based on the preset time window sliding, the access times of the stripe are extracted;

[0013] The access frequency is determined by performing a ratio operation on the access times and the window length of the time window.

[0014] Optionally, for stripes whose current configuration version number's heat level is inconsistent with the associated current stripe heat level, updating the current configuration version number and the locally repairable code stripe configuration parameters based on the corresponding current stripe heat level includes:

[0015] For stripes whose current configuration version number's heat level is inconsistent with the associated current stripe heat level, match the stripe configuration parameter threshold based on the corresponding current stripe heat level;

[0016] Adjust the local repairable code stripe configuration parameters of the corresponding stripe according to the threshold values ​​of each stripe configuration parameter, and update the current configuration version number.

[0017] Optionally, the step of traversing the clusters of the distributed storage system and determining the priority scheduling clusters for each of the changed data blocks includes:

[0018] The clusters of the distributed storage system are traversed, and the scheduling priority of each cluster for each changed data block is calculated based on the bandwidth of the changed data block to the cluster and whether the changed data block and its stripe are in the same cluster.

[0019] The cluster with the highest scheduling priority to which each of the modified data blocks belongs is designated as the corresponding priority scheduling cluster.

[0020] Optionally, the step of writing each of the changed data blocks and the corresponding local parity block to the corresponding priority scheduling cluster and updating the corresponding stripes includes:

[0021] The target write node for each priority scheduling cluster is determined based on the storage available capacity ratio and write path available bandwidth of each storage node in each priority scheduling cluster.

[0022] If the load ratio of the distributed storage system is not greater than the high load threshold, then the associated changed data block and local parity block are written to the corresponding priority scheduling cluster according to each target write node, and the corresponding stripes are updated.

[0023] Optionally, it also includes:

[0024] Real-time monitoring of the average storage availability ratio and load ratio of the distributed storage system;

[0025] The adjustment strategy for the local repairable code stripe configuration parameters of each stripe is determined based on the average storage availability ratio and the load ratio.

[0026] A second aspect of the present invention provides a dynamically configured distributed storage optimization device, applied to a distributed storage system, the device comprising:

[0027] The access behavior detection module is configured to collect access frequencies of each stripe in the distributed storage system in real time, and determine current stripe heat levels of each stripe based on the access frequencies.

[0028] The heat parameter updating module is configured to update the current configuration version number and the local repairable code stripe configuration parameter based on the corresponding current stripe heat level for a stripe whose current stripe heat level is inconsistent with the stripe heat level of the current configuration version number.

[0029] The incremental data identification module is configured to identify incremental data in each stripe in real time, and determine changed data blocks.

[0030] The scheduling cluster determination module is configured to traverse clusters of the distributed storage system, and determine priority scheduling clusters of each changed data block.

[0031] The check block re-encoding module is configured to perform local re-encoding on each changed data block, and update local parity check blocks associated with each changed data block.

[0032] The data writing updating module is configured to write each changed data block and the corresponding local parity check block into the corresponding priority scheduling cluster, and update the corresponding stripe.

[0033] The third aspect of the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the distributed storage optimization method based on dynamic configuration according to any one of the above aspects.

[0034] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the distributed storage optimization method based on dynamic configuration according to any one of the above aspects.

[0035] The fifth aspect of the present application provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the distributed storage optimization method based on dynamic configuration according to any one of the above aspects.

[0036] From the above technical solutions, the present application has the following advantages:

[0037] The above-described solution of the present invention provides a distributed storage optimization method based on dynamic configuration. The method includes: real-time acquisition of the access frequency of each stripe in the distributed storage system, and determining the current stripe heat level based on each access frequency; for stripes whose current configuration version number's stripe heat level is inconsistent with the associated current stripe heat level, updating the current configuration version number and locally repairable code stripe configuration parameters based on the corresponding current stripe heat level; real-time incremental data identification in each stripe to determine changed data blocks; traversing the cluster of the distributed storage system to determine the priority scheduling cluster for each changed data block; performing local recoding on each changed data block and updating the local parity block associated with each changed data block; writing each changed data block and its corresponding local parity block to the corresponding priority scheduling cluster and updating the corresponding stripe. Based on the above solution, by dynamically adjusting the locally repairable code stripe configuration parameters by sensing access frequency, performing local recoding based on incremental data identification, and traversing the cluster to optimize scheduling data, the overall data storage reliability of the distributed storage system is improved. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 An example diagram illustrating the structural composition of an LRC provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the LRC cluster storage architecture provided in an embodiment of the present invention;

[0041] Figure 3 A flowchart illustrating the steps of a distributed storage optimization method based on dynamic configuration, as provided in an embodiment of the present invention.

[0042] Figure 4 A flowchart illustrating a distributed storage optimization method based on dynamic configuration, provided in an embodiment of the present invention.

[0043] Figure 5 This is a structural block diagram of a dynamically configured distributed storage optimization device provided in an embodiment of the present invention. Detailed Implementation

[0044] This invention provides a method and apparatus for optimizing distributed storage based on dynamic configuration, which can improve the data storage reliability of distributed storage systems.

[0045] In order to make the application purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings of the embodiments of the present application. Obviously, the following described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0046] Term explanation:

[0047] Distributed storage system: In a typical distributed storage system, data is split and stored in multiple computing nodes to achieve high availability and fault tolerance; each node can act as a data block storage and also undertake the calculation task of a check block; nodes are interconnected through high-speed networks to form a logically unified distributed file system; this distributed node storage model supports horizontal expansion and elastic capacity management, and is the basic architecture form of modern object storage, block storage and file storage services.

[0048] Striping storage mechanism: striping is a mechanism that divides file data into multiple fixed-size data blocks and stores them on multiple nodes according to certain rules; this mechanism not only improves parallel read-write performance, but also provides a structural basis for redundancy coding (such as erasure coding); each group of striping usually includes data blocks and redundancy blocks (check blocks), which improve system fault tolerance through distributed redundancy strategy, and when a node fails, data can be recovered through blocks in other striping.

[0049] Locally repairable code (LRC): With the explosive growth of data, modern distributed storage systems widely use erasure coding (error correction code) to improve storage efficiency and fault tolerance; compared with replication mechanism, erasure coding divides data into multiple data blocks and parity blocks to form a striping, achieving higher storage utilization; however, traditional erasure codes, such as Reed-Solomon code, usually need to read multiple data blocks from the entire data striping across nodes when repairing a single data block, resulting in high bandwidth and computing overhead;

[0050] Local repair code is an optimized erasure code structure, usually composed of multiple local groups, each containing several data blocks and a local parity block. By grouping data blocks and introducing local repair mechanism, it significantly reduces the number of blocks required for repair and the amount of cross-node access while ensuring fault tolerance, thereby improving repair efficiency and reducing resource consumption. It is particularly suitable for large-scale stripe merging and dynamic repair scenarios. In addition, global parity blocks can be configured to enhance the overall robustness of the system. LRC is widely used in practical systems (such as Azure Storage) to effectively solve the high reconstruction cost problem of classic Reed-Solomon encoding in the repair process. In implementation, the structure of LRC can be configured using three parameters , where is the number of data blocks, which can be represented as , used as the basic storage unit for user data, and the parameter is the number of local parity blocks, which are represented as , each of which is generated by encoding a subset of data blocks within its local group to achieve efficient local repair, and the parameter is the number of global parity blocks, which are represented as , calculated from all data blocks to provide global fault tolerance. Therefore, an LRC stripe is composed of data and parity blocks, which is the basic unit in a distributed storage system, Figure 1 shows an example of LRC with ;

[0051] Figure 2 shows a cluster storage architecture based on LRC, where data and parity blocks are distributed across three clusters, each with four storage nodes. Although intra-cluster communication enjoys high bandwidth for efficient local repair, inter-cluster bandwidth is limited by the core network, resulting in significant performance overhead during data repair and stripe merging operations. In practical applications, the high cost of inter-cluster communication significantly affects the efficiency of data repair and stripe merging. For example, data migration and parity block recalculation during stripe merging can cause significant network load. Therefore, how to optimize bandwidth usage in a distributed storage architecture to improve system performance has become an important research direction.

[0052] Stripe merging: In distributed systems with LRC, stripe merging is a key operation in the process of reconstruction and data management, especially when data migration, capacity rebalancing or node replacement are frequent; stripe merging requires recalculation of parity blocks, usually involving data transmission between multiple nodes, resulting in high bandwidth consumption and system load; in addition, data consistency and fault tolerance need to be maintained during the merging process, increasing scheduling complexity and metadata management overhead;

[0053] Therefore, during the generation of wide stripes, the distributed storage system needs to solve several technical challenges, including data migration overhead, inter-cluster communication traffic, dynamic adjustment of stripe parameters, optimization of data placement strategies, and fault isolation, which directly affect system efficiency, fault tolerance and overall performance. During the generation of wide stripes, data migration constitutes an important part of network overhead, especially during stripe merging, the total data migration cost can be mathematically modeled as follows:

[0054] ;

[0055] wherein, is the total data migration cost, is the block index, is the total number of data blocks being migrated, is the th data block, is the size of the data block;

[0056] This migration process is crucial for redistributing data blocks across clusters to ensure balanced storage and fault tolerance, but such operations can cause a large amount of cross-cluster traffic and performance degradation. To alleviate these problems, a phased stripe merging is proposed as an effective strategy, in which stripe merging is performed in multiple stages in an incremental manner, thereby reducing the amount of instantaneous migration. The phased data migration cost can be expressed as:

[0057] ;

[0058] wherein, is the phased data migration cost, is the block index, is the merging stage index, is the number of stripes merged in the th stage, is the total number of stages, is the th data block merged in the th stage, is the size of the data block;

[0059] To reduce the system overhead of the merging process, the embodiment proposes an incremental staged merging model (ISMM), inspired by the multi-round merging technique, ISMM introduces selective recomputation and partial propagation, through local update and phased processing strategy, in each phase, only the updated part of the stripe will be merged and re-encoded, by avoiding full stripe recomputation and unnecessary data migration, significantly reducing the amount of data that needs to be migrated and the number of redundant calculations, in addition, the layout-aware design can ensure that the merging operation respects data locality and maximizes the reduction of cross-cluster traffic, thereby improving efficiency and scalability, at the same time, the model supports dynamic adjustment of stripe parameters (such as the number of data blocks , the number of local parity blocks ) according to the cluster load, reducing the repair cost in high-load scenarios, and improving resource utilization in low-load scenarios; due to the limited inter-cluster bandwidth relative to the high throughput within the cluster, inter-cluster communication becomes the main bottleneck, recomputing global parity or cross-cluster merging stripes involves inter-cluster data transmission, which will generate a large amount of network traffic, which can be quantitatively modeled to guide the optimization of merging efficiency and bandwidth consumption.

[0060] Please refer to Figure 3 and Figure 4 , the embodiment provides a distributed storage optimization method based on dynamic configuration, applied to a distributed storage system, the method comprises:

[0061] Step 101, real-time collection of access frequency of each stripe in the distributed storage system, and determination of current stripe heat level of each stripe based on the access frequency.

[0062] Access frequency refers to the number of accesses to the stripe in a unit of time. The number of accesses refers to the number of read and write operations of data.

[0063] Stripe heat level refers to the classification level of stripe access heat. Current refers to the state corresponding to the latest time point.

[0064] It should be noted that the embodiment real-time monitors the access frequency of each stripe (Stripe) in the distributed storage system, and also supports writing the access frequency data into the metadata table of the distributed storage system. The metadata table is a structured data table for storing metadata, which can be used to record attribute information of data for subsequent heat identification. Then, the access heat is determined based on the dynamic matching of the stripe heat level based on the access frequency.

[0065] In one specific embodiment of the embodiment, the determination process of the access frequency comprises:

[0066] The access times of the strip are extracted based on a preset time window sliding;

[0067] The access frequency is determined by ratio operation of the access times and the window length of the time window.

[0068] It should be noted that in the embodiment, the access frequency of each strip is continuously counted by sliding time window, and the calculation formula is as follows:

[0069] ;

[0070] In the formula, is the strip, is the access frequency of the strip, is the window length of the time window, is the access times of the strip; the sliding window mechanism is used to capture the access time locality, which helps to improve the real-time performance and accuracy of the hotness evaluation.

[0071] In one specific embodiment of the embodiment, the determination process of the strip hotness level includes: arranging the access frequencies in ascending order to determine a frequency sequence; determining a hotness threshold from the frequency sequence based on quantiles, and dividing the strip hotness level according to the hotness threshold.

[0072] The frequency sequence refers to an ordered sequence formed by arranging the access frequencies in a specific order.

[0073] The hotness threshold refers to a critical value for dividing the strip hotness level.

[0074] It should be noted that the embodiment dynamically calculates the hotness threshold based on quantiles, which can better adapt to changes in data access patterns; for example, the hotness threshold set by the embodiment includes a hot data threshold and a warm data threshold According to the access frequency, the strip is divided into three strip hotness levels: high-heat strip (Hot), medium-heat strip (Warm), and low-heat strip (Cold). wherein, ≈80% quantile of the frequency sequence, ≈50% quantile of the frequency sequence.

[0075] Step 102, for the strip whose strip hotness level of the current configuration version number is inconsistent with the associated current strip hotness level, updating the current configuration version number and the local repairable code strip configuration parameter based on the corresponding current strip hotness level.

[0076] ​​The configuration version number refers to an identifier marking the version of the local repairable code strip configuration parameter used by the strip. The current refers to the state corresponding to the latest time point.

[0077] The local repairable code strip configuration parameter includes the number of data blocks, the number of local parity check blocks, and the number of global parity check blocks, which defines the LRC configuration structure in the strip.

[0078] It should be noted that since the existing system generally fails to adaptively adjust the strip structure according to the access heat of data (hot / warm / cold), the repair efficiency of hot data in a wide strip is low, and cold data occupies too much redundant resources in a narrow strip, and there are limitations such as suboptimal resource utilization and repair efficiency. Therefore, the present embodiment considers to adjust the parameters in a perception manner, adaptively adjusts the strip width based on the strip heat level, so that the strip configuration matches the actual access characteristics, reduces redundancy or calculation waste, and optimizes the storage efficiency;

[0079] In a specific implementation, an inert update strategy is adopted, and new configuration is written only when the strip heat level changes, that is, if the current configuration version number corresponds to a strip heat level different from the current strip heat level, the local repairable code strip configuration parameter in the metadata is updated, such as and the current configuration version number ← , so as to maintain the consistency of the strip structure and the system access characteristics, and also serve as the basis for subsequent difference merging scheduling.

[0080] It can be understood that the number of global parity check blocks can be determined based on the number of data blocks and the number of local parity check blocks, and the specific determination mechanism can refer to the prior art. Therefore, when adjusting according to the heat, the number of data blocks and the number of local parity check blocks can be adjusted according to the strip heat level, and the number of global parity check blocks can be adaptively changed with the changes of the two numbers.

[0081] In one specific embodiment of the present embodiment, step 102 includes the following sub-steps:

[0082] For the strip whose current configuration version number and associated current strip heat level are inconsistent, the strip configuration parameter threshold value is matched based on the corresponding current strip heat level.

[0083] According to the strip configuration parameter threshold value, the local repairable code strip configuration parameter of the corresponding strip is adjusted, and the current configuration version number is updated.

[0084] The strip configuration parameter threshold value refers to a set of pre-set reference values or reference ranges of the number of data blocks and the number of local parity check blocks.

[0085] It should be noted that in this embodiment, the number of data blocks and local parity blocks are allocated according to the stripe heat level. This configuration relationship is characterized by the stripe configuration parameter threshold. For example, high-heat stripe (Hot) → and Warm strip → and Cold strip → and Therefore, if a change in the stripe heat level is detected, the local repairable code stripe configuration parameters are adjusted according to the stripe configuration parameter threshold corresponding to the current stripe heat level, and the current configuration version number is updated to indicate the configuration change status.

[0086] Step 103: Incremental data identification is performed on each strip in real time to determine the changed data blocks.

[0087] Incremental data refers to the portion of data that has been modified or added relative to the historical state.

[0088] A modified data block refers to a data block whose data content has been modified or added.

[0089] It should be noted that a certain scheduling period can be set to identify each stripe in the current scheduling period. The data blocks that have been modified or added within the data block are used to determine the incremental data block set. , For block indexes, For the first One data block, For the first One changed data block, Striping significantly reduces the scope of subsequent recoding and is a key input for incremental merging. In practical implementation, incremental data identification methods can include: determining data blocks with write records as changed data blocks based on the write log of the distributed storage system; determining changed data blocks with incremental data through hash comparison based on hash digests; filtering changed data blocks with incremental data based on Bloom filters; or locating changed data blocks based on a combination of write logs and hash digests, Bloom filters and hash digests, write logs and Bloom filters, or a combination of write logs, Bloom filters, and hash digests. The specific method can be flexibly selected based on the application scenario. It is understood that the write log can be a system-level write log, or a node-level or block-level write log.

[0090] Step 104: Traverse the clusters of the distributed storage system and determine the priority scheduling cluster for each changed data block.

[0091] The priority scheduling cluster refers to a cluster more suitable for merging the change data block.

[0092] It should be noted that the embodiment adopts incremental merging. Since the migration of the change data block can involve cross-cluster communication, in order to reduce the cross-cluster communication load, a suitable priority scheduling cluster needs to be selected for each change data block in each cluster of the distributed storage system.

[0093] In one specific embodiment of the embodiment, step 104 includes the following sub-steps:

[0094] Traverse the clusters of the distributed storage system, and calculate the scheduling priority of each cluster for each change data block based on the bandwidth of the cluster to which the change data block goes and whether the change data block and the belonging stripe are located in the same cluster;

[0095] The cluster with the maximum scheduling priority of each change data block is taken as the corresponding priority scheduling cluster.

[0096] The scheduling priority refers to a quantitative score of determining the amount of change data block merging performed by each cluster.

[0097] It should be noted that when determining the priority scheduling cluster, it is considered that the merging operation is preferentially completed within the same cluster, and the cross-cluster communication adopts path bandwidth optimization scheduling to improve network utilization and reduce bottlenecks; in specific implementation, the clusters of the distributed storage system are traversed, it is identified whether the change data block involves cross-cluster transmission, and on this basis, the scheduling priority of each cluster is determined by the intra-cluster priority and bandwidth-aware path selection, avoiding the network overhead caused by cross-cluster redundant reconstruction, and realizing bandwidth-aware scheduling.

[0098] The calculation process of the scheduling priority includes: , is the cluster index, is the th cluster, is the scheduling priority, is the decision of whether the change data block and the belonging stripe are located in the same cluster, indicates that the change data block and the belonging stripe are located in the same cluster, indicates that the change data block and the belonging stripe are not located in the same cluster, is the bandwidth to the cluster (i.e. the bandwidth of moving the change data block from the current location to the cluster ), is the same cluster decision coefficient, is the bandwidth coefficient; for example, if a change data block belongs to , and the block structure of the stripe is distributed in the cluster and , the changed data block and the stripe to which the changed data block belongs are located in the same cluster, that is, at this time , , the changed data block and the stripe to which the changed data block belongs are not located in the same cluster, that is, at this time , , the changed data block and the stripe to which the changed data block belongs are not located in the same cluster, that is, at this time , In a specific implementation, may be set to 0.7, may be set to 0.3.

[0099] Step 105, performing local recoding on each changed data block to update the local parity block associated with each changed data block.

[0100] It should be noted that, unlike the case that the traditional full-stripe reconstruction strategy leads to huge computing overhead, the embodiment only recalculates and updates the local parity block related to the changed data block, which helps to greatly reduce the consumption of computing resources and improve the merging concurrency and processing efficiency; in a specific implementation, for each , a local coding function is used to perform local recoding to generate a corresponding updated local parity block .

[0101] Step 106, writing each changed data block and the corresponding local parity block to the corresponding priority scheduling cluster, and updating the corresponding stripe.

[0102] In one specific implementation of the embodiment, step 106 includes the following sub-steps:

[0103] Writing each changed data block and the corresponding local parity block to the corresponding priority scheduling cluster includes:

[0104] Determining the target write node of each priority scheduling cluster according to the storage available capacity ratio of each storage node in the priority scheduling cluster and the available bandwidth of the write path;

[0105] If the load ratio of the distributed storage system is not greater than the high load threshold, write the associated changed data block and local parity block to the corresponding priority scheduling cluster according to each target write node, and update the corresponding stripe.

[0106] The target write node refers to the storage node to which the updated block is written.

[0107] The high load threshold refers to a threshold representing that the system is in a high load state.

[0108] It should be noted that in this embodiment, the scheduler in the distributed storage system makes a combined decision based on the storage available capacity ratio of the storage node and the available bandwidth and load ratio of the write path to complete the writing of the modified data block and the local parity block.

[0109] The storage available capacity ratio of a storage node is a metric that measures the remaining capacity of the node. The calculation process includes:

[0110] ;

[0111] In the formula, For storing node indexes, For the first The ratio of available storage capacity per storage node For the first Available storage capacity of each storage node For the first Total storage capacity of each storage node;

[0112] The available bandwidth of the write path refers to the bandwidth currently available on the transmission path for writing changed data blocks to another storage node. This embodiment considers a bandwidth priority and storage balance decision strategy, and determines the target write node by comprehensively considering the storage available capacity ratio and the available bandwidth of the write path to ensure system stability. In one implementation, the write index value can be determined by weighting the storage available capacity ratio and the available bandwidth of the write path, and the storage node with the largest write index value can be selected as the target write node. In this case, the weight of the storage available capacity ratio is less than the weight of the available bandwidth of the write path. In another implementation, the storage node with the largest available bandwidth of the write path is first determined as a candidate write node. If there is only one candidate write node, it is selected as the target write node. If there are multiple candidate write nodes, the candidate write node with the largest storage available capacity ratio is selected as the target write node.

[0113] Load ratio (LR) is used to measure the current operating load of a system and is defined as follows:

[0114] ;

[0115] In the formula, For the current input / output workload, This represents the system's maximum processing capacity.

[0116] For the load ratio, this embodiment sets a high load threshold. If the current load ratio is greater than the high load threshold, it means that the I / O processing capacity of the system is close to saturation. At this time, writing should be avoided to prevent I / O congestion. Until the current load ratio is not greater than the high load threshold, the corresponding changed data blocks and local parity blocks are written according to each target write node, and the corresponding stripes are updated.

[0117] To better illustrate, referring to Table 1, the steps 101-106 performed by the embodiment one of the present application are exemplified in the form of algorithm steps:

[0118] To provide a structured approach to reduce cross-cluster traffic in an incremental manner, Algorithm 1 proposes an incremental traffic optimization in the strip merging mechanism, dynamically adjusting the strip width to enhance storage system performance while balancing storage resource utilization and repair cost throughout the data life cycle:

[0119] Initially, the system initializes the strip merging process (line 1) to ensure that the merging operation is performed in a controlled environment; at step 1 (lines 2-13), the system adaptively adjusts the strip width based on the data access pattern to optimize storage efficiency; when all the strips (line 3) are traversed, the system calculates their access frequencies (line 4) and assigns different strip width parameters according to high, medium, and low access levels, i.e., strip hotness levels (lines 5-10), where high access frequency data utilizes narrow strips (smaller ) to reduce local repair computation overhead (lines 5-6), medium access frequency data uses medium strip width (lines 7-8), and low access frequency data utilizes wide strips (larger ) to improve storage resource utilization and minimize global redundancy overhead (lines 9-10); after adjustment, the system updates the strip configuration (line 12) to ensure that the strip structure meets the storage requirements;

[0120] At step 2 (lines 14-22), the system performs incremental data migration and repair optimization to further reduce cross-cluster data transmission; after traversing all the strips (line 15), the system identifies the affected storage cluster Caffected (line 16), i.e., the priority scheduling cluster, which contains data blocks that need to be repaired or migrated, i.e., changed data blocks; for each affected cluster (line 17), the system retrieves the data blocks (line 18), recalculates the local parity blocks (line 19), and stores the optimized strips (line 20);

[0121] This incremental repair strategy reduces the frequency of global parity block recalculation, reduces cross-cluster repair cost, and enhances the reliability and scalability of the storage system (lines 21-22).

[0122] It should be noted that only a brief description is made here, and the specific implementation process of each step can be understood by referring to the related content in the foregoing embodiments, which will not be described here. It can be understood that the present application does not limit this.

[0123] Table 1 Algorithm 1 adaptive incremental strip merging and flow optimization

[0124]

[0125] To further improve the parameter configuration strategy in "Algorithm 1 adaptive incremental strip merging and flow optimization", the present embodiment details the recommended range, suggested configuration, configuration basis and typical configuration example of the key parameters of strip configuration, including the number of data blocks , the number of local redundancies and the number of global redundancies , to ensure that the model has good adjustability and practicability under different loads and access modes, thereby improving the scalability, fault tolerance and resource utilization efficiency of the system:

[0126] First, the number of original data blocks in the strip , the recommended value range is generally 4 to 10; a smaller value is suitable for high-frequency access data, at which time a narrower strip configuration can significantly reduce the calculation overhead of local repair; while a larger value is suitable for low-frequency archival data, which helps to improve the overall storage utilization and reduce the recalculation frequency of global parity blocks; in the dynamic adjustment strategy of the present model, , and represent the recommended configurations for high, medium and low access frequency data, i.e. different strip heat levels. Exemplarily, the specific suggestions can be: , , ;

[0127] Secondly, regarding the number of local parity blocks , which is mainly used to support local fault tolerance and fast repair operations; the recommended value of l is generally 1 to 3; for small-scale strips, l can be configured to reduce the redundancy calculation cost; for medium-scale strips, l is recommended to be to balance the local repair efficiency and storage overhead; while in large-scale strips (such as ), l can be used to cope with higher data block density and potential failure probability; simulation experiments show that is the optimal choice in most general scenarios, which can balance repair efficiency and bandwidth consumption;

[0128] For the number of global parity blocks A value of 2 to 4 is recommended; higher values ​​are also acceptable. This value can significantly improve the system's fault tolerance under multi-node failures, making it suitable for mission-critical storage scenarios; however, in scenarios where repair efficiency is the priority, The value can be appropriately reduced to 2 to decrease cross-cluster communication traffic and global recalculation overhead. Considering that the ISMM model adopts incremental merging and local optimization mechanisms, which can effectively reduce the frequency of global recalculation, it is recommended to use 2 in most scenarios. A configuration of 2 or 3 is chosen as a trade-off between redundancy and performance.

[0129] Based on the above parameters, this embodiment proposes the following stripe configuration to guide actual deployment:

[0130] 1. Hot Data (High-Frequency Access): 1) Conditions: High access frequency, frequent updates, latency sensitive; 2) Configuration parameters: , , 3) Configuration effect: The narrow stripe structure reduces local repair overhead and redundant calculations, making it suitable for data scenarios requiring rapid response; 4) Merging strategy: Only incremental data ∆D is partially reconstructed, and merging tasks are completed on local nodes first to avoid cross-cluster communication; 2. Warm data (medium-frequency access): 1) Conditions: Moderate access frequency and balanced read and write operations; 2) Configuration parameters: , , ;3) Configuration effect: It maintains a balance between redundancy efficiency and repair performance, and is suitable for data caching or periodic write scenarios;4) Merging strategy: It combines access popularity and topology scheduling priority function to execute merging tasks on paths with sufficient bandwidth and appropriately enables cross-cluster scheduling.

[0131] 3. Cold Data (Low-Frequency Access): 1) Conditions: No access after writing, high tolerance for repair; 2) Configuration Parameters: , , 3) Configuration effect: The wide stripe structure improves storage density and enhances fault tolerance, making it suitable for archiving, cold backup and other scenarios; 4) Merging strategy: Merging tasks are concentrated in batches during low-load periods, supporting cross-cluster scheduling and bandwidth-aware path optimization.

[0132] In summary, the locally repairable code stripe configuration parameters proposed in this embodiment can provide a theoretical basis and practical guidance for "Algorithm 1 Adaptive Incremental Stripe Merging and Traffic Optimization". Through a dynamic adjustment mechanism, the adaptability and robustness of the model in actual distributed storage systems are enhanced, effectively supporting an efficient and scalable cross-cluster storage management system.

[0133] In one specific implementation of the embodiment, further comprising: detecting the average storage availability ratio and the load ratio of the distributed storage system in real time; and determining the adjustment strategy of the local repairable code strip configuration parameters of each strip according to the average storage availability ratio and the load ratio.

[0134] The adjustment strategy includes reducing or increasing or not modifying the number of data blocks and the number of local parity blocks.

[0135] It should be noted that the traditional fixed-width strip design uses static parameters (data blocks) and (local parity blocks), which remain unchanged throughout the system. Although such a configuration can maintain balance under stable workloads, they are not suitable for dynamic environments with fluctuating access patterns and heterogeneous resource availability. These limitations can lead to low storage utilization, increased computational overhead, and increased cross-cluster data movement; To solve this problem, the ISMM mechanism proposed in the embodiment not only adjusts the strip width according to the access frequency, but also adaptively adjusts the strip width based on workload characteristics and storage conditions, prioritizing consolidation and repair tasks between nodes with ample bandwidth and high data locality, reducing cross-cluster communication overhead. In addition, the strategy supports cost-oriented scheduling optimization, such as performing high-overhead tasks during off-peak hours to achieve temporal and spatial decoupling of system resources and maximize global efficiency.

[0136] For storage conditions, the average storage availability ratio is introduced as an important indicator to measure the remaining capacity of the entire distributed storage system, defined as the average of all storage node availability ratios:

[0137]

[0138] where is the storage node index, is the storage availability ratio of the th storage node, is the total number of storage nodes, is the th average storage availability ratio.

[0139] For the average storage availability ratio, set a high storage pressure threshold (such as the range: 0.2-0.3) and a low storage pressure threshold (such as the range: 0.5-0.6), when , it means that most nodes in the system are close to the capacity limit and are in a high-pressure state, when , it means that the system has ample space to accommodate a higher redundancy ratio; According to the above threshold representation, the system scheduling strategy is as follows:

[0140] 1. If​ High pressure: 1) Reduce the number of local parity blocks , release available storage space; 2) Reduce the number of data blocks , narrow the strip width, and distribute data more granularly across multiple cluster nodes to alleviate single-node storage pressure; 3) Preferentially select nodes with higher storage availability to perform merging or redundant writing to avoid write amplification or overflow risk on hot nodes; 4) Postpone cross-cluster merging operations and prioritize local repair within the available resource range;

[0141] 2、If Low pressure: 1) Increase , improve local repair capability, and reduce dependence on global parity block recalculation; 2) Increase , widen the strip, improve storage utilization, and reduce redundancy ratio; 3) In the merging task scheduling, preferentially arrange high-redundancy strips for merging to recover storage resources with excessively high redundancy; 4) Allow the system to perform global merging operations under idle bandwidth to improve system consistency and long-term reliability;

[0142] In addition to storage resource status, workload characteristics are also important factors affecting strip parameter adjustment. The load of a storage system is usually determined by data reading, writing, and strip merging operations. Therefore, consider optimizing the values of and based on cluster load conditions, with Load Ratio (LR) as a measure of the current system load state;

[0143] For Load Ratio, set a high load threshold such as range: 0.8-0.9) and a low load threshold such as range: 0.3-0.4), if , the system is high load, and if , the system is idle or in a low activity state; based on the Load Ratio, the cluster scheduling strategy is as follows:

[0144] 1、When High load: 1) Reduce , use narrow strip configuration to reduce the number of data blocks involved in single read / write or repair, and reduce access conflicts; 2) Reduce , reduce local redundancy calculation and write operation amount during writing; 3) The scheduler delays merging tasks to low peak periods or performs batch local merging to avoid system resource competition; 4) Prohibit or limit cross-cluster data migration and merging operations to prioritize real-time business I / O performance;

[0145] 2、When Low load: 1) Increase , expand strip width, improve overall data throughput; 2) increase , strengthen local fault tolerance, reduce subsequent global repair pressure; 3) the scheduler captures the load empty window period, and concentrates on executing high computation or high communication traffic strip reconstruction and global merging tasks; 4) start the bandwidth utilization optimization module to improve resource use efficiency;

[0146] In actual deployment, the system jointly judges and State, select whether to perform strip merging, how to adjust the strip structure and how to schedule data migration tasks, for example:

[0147] 1, when and (high storage pressure + high load): the system enters "conservative mode", prohibits merging, compression parameters and , limit writing, priority to guarantee business I / O performance;

[0148] 2, when and (storage tension + medium / low load): the system enters "storage control mode", moderately reduces the redundancy parameter (reduces or ), speed up the cold data strip merging and archiving, release storage space, high load tasks will be delayed;

[0149] 3, when and (resource abundant + low load): the system enters "optimization mode", starts batch execution of redundancy merging and reconstruction operation to recover redundant resources, and improves storage efficiency;

[0150] 4, when and (resource abundant + medium / high load): the system enables "load-aware mode", maintains a wider strip configuration (larger , ), but the merging task scheduling selects the low load path or is scheduled to low peak execution to avoid real-time business congestion;

[0151] It can be understood that when the resource balance state, that is, the average storage availability of the system is in the middle threshold interval, it indicates that the overall storage resource use of the system is balanced, and there is no excessive tension or redundant waste;

[0152] 5, and (In resource and load balancing state): The system maintains "stable mode," keeping the current stripe configuration unchanged and triggering local merging tasks only based on the ∆D increment; merging scheduling follows the "topology awareness + bandwidth priority" principle, prioritizing execution in low-latency, low-load paths; in this state, the system aims to maintain operational stability and avoid resource jitter caused by frequent parameter fluctuations.

[0153] 6. When and When resources are balanced and the load is low, the system enters "performance optimization mode". At this time, the performance can be improved by increasing the stripe width (increasing...). ) and the number of local redundant blocks (increase) To improve storage utilization and fault tolerance, the system will concentrate on executing computationally intensive stripe reconstruction and global merging tasks during idle time windows to make full use of bandwidth and computing power. This strategy can effectively reduce resource waste caused by excessive redundancy or stripe fragmentation during long-term system operation.

[0154] 7. When and When resources are balanced and under high load, the system enters "load mitigation mode." Although storage resources are sufficient, in order to ensure read and write performance and reduce computational pressure, it will actively reduce the stripe width (reduce the number of data blocks). ) and the number of local redundancies (reducing) This reduces the number of data blocks involved in a single merge or repair operation, thereby reducing I / O conflicts and computational resource consumption. At the same time, the system scheduler will delay global merge tasks for non-critical stripes, retaining only necessary local repair operations. Cross-cluster merge operations will be restricted or queued to prioritize low-latency processing of real-time services.

[0155] In summary, by " "and" "By incorporating the scheduling decision-making system, the ISMM mechanism can achieve dual awareness of storage capacity and system load status, and dynamically adjust stripe parameters and optimize cluster task scheduling accordingly. The system supports dynamic switching between distributed and centralized redundant layouts, and automatically adjusts stripe positions and layout structures when storage node load is too high or cross-cluster communication is unreachable, thereby maximizing bandwidth utilization and balancing storage resources, thus maximizing resource utilization and improving the overall system operating efficiency and scalability."

[0156] In one specific implementation of this embodiment, the method further includes determining the total merger cost based on a merger cost function: , To consolidate total costs, The computational overhead (which mainly comes from the reconstruction operation of the local parity block involved in the changed data block) is the main source of the computational overhead. The communication cost depends on the number of cross-cluster transport blocks and path bandwidth. For storage costs (considering) (Compared to strip redundancy ratio) As the first configurable weight, As the second configurable weight, For the third configurable weight, such as setting , and The computational overhead, communication costs, and storage costs can be determined by combining existing technologies. In one implementation, , For block indexes, To change the number of data blocks, For the first One changed data block, For the size of the data block, Available bandwidth for the write path;

[0157] Furthermore, parameters can be fine-tuned based on the total merger cost. This parameter tuning can target any adjustable threshold, weight, or coefficient from the preceding steps, such as in the next round. , By configuring or scheduling priority coefficients, and combining steps 101-106, a closed-loop strategy process of "access awareness—configuration adjustment—incremental identification—topology scheduling—local reconstruction—cost feedback" is formed. Under this simulation and visualization analysis framework, key cost indicators under different merging strategies can be modeled and quantified. This framework can support simulation input of various coding configurations and stripe layout modes. It can predict data transmission volume, node load changes, and coding overhead curves in advance before merging optimization, and present resource trends, bottlenecks, and performance gains during the merging process through a graphical interface. This enhances the system's deployment flexibility and operational controllability, and has high professionalism, engineering feasibility, and system stability. It is suitable for stripe merging optimization tasks in large-scale distributed erase coding storage systems under dynamic load scenarios.

[0158] To better illustrate this, a simplified calculation example is provided:

[0159] 1. System background premise: 1) Stripes The original configuration was ( , , Access frequency 3 times / minute (cold data); 2) Current scheduling cycle Within, the frequency of access increased to It is 9 times per minute, which is more than 3) System , It is in a state of medium load and moderate storage resources; 4) Two data blocks , Updated, constitute 5) The storage system includes a cluster. , and ,current exist , exist Bandwidth measurement , , , ;

[0160] 2. Processing flow:

[0161] Step 1: Detection Increased access frequency → →The heat level of the strip changes from Cold to Hot;

[0162] Step 2: Temperature change triggers parameter reconfiguration: ←5, ←1 (corresponding to thermal data configuration);

[0163] Step 3: Update Configuration: { , , }, ← ;

[0164] Step 4: Identify Record the cluster to which it belongs;

[0165] Step 5: Calculate scheduling priority , Preferred Prioritize scheduling clusters;

[0166] Step 6: [To] , Line-local recoding, generating Update the local parity block;

[0167] Step 7: Write to target middle Storage nodes, bandwidth path Write successful;

[0168] Step 8: Consolidate the evaluation and record the cost indicators for subsequent optimization.

[0169] In the embodiment of the present application, a comprehensive optimization framework is proposed to improve the data storage reliability of the distributed storage system, 1) a dynamic strip width adjustment mechanism is introduced, which realizes more efficient resource supply and improves the system response ability under heterogeneous conditions; 2) an incremental strip merging scheme based on data change is proposed, which significantly reduces the consumption of computing resources and coding delay; 3) a topology-aware data scheduling and cross-cluster communication optimization mechanism is designed, which preferentially performs merging and repair tasks on the same cluster or bandwidth-rich path, thereby effectively avoiding network congestion and cross-cluster bandwidth waste; 4) a visual merging cost evaluation and strategy optimization scheme is constructed, which is helpful to realize fine management and strategy optimization of the strip layout scheme.

[0170] Please refer to Figure 5 , Figure 5 The structure block diagram of a distributed storage optimization device based on dynamic configuration provided by the embodiment of the present application.

[0171] The distributed storage optimization device based on dynamic configuration provided by the embodiment of the present application is applied to a distributed storage system, and the device comprises:

[0172] The access behavior detection module 501 is configured to collect the access frequencies of the strips in the distributed storage system in real time, and determine the current strip heat levels of the strips based on the access frequencies.

[0173] The heat parameter updating module 502 is configured to update the current configuration version number and the local repairable code strip configuration parameters based on the corresponding current strip heat levels for the strips whose current strip heat levels are inconsistent with the strip heat levels of the current configuration version number.

[0174] The incremental data identification module 503 is configured to identify the incremental data in the strips in real time, and determine the changed data blocks.

[0175] The scheduling cluster determination module 504 is configured to traverse the clusters of the distributed storage system, and determine the priority scheduling clusters of the changed data blocks, respectively.

[0176] The check block recoding module 505 is configured to perform local recoding on the changed data blocks, and update the local parity check blocks associated with the changed data blocks.

[0177] The data writing updating module 506 is configured to write the changed data blocks and the corresponding local parity check blocks into the corresponding priority scheduling clusters, and update the corresponding strips.

[0178] Optionally, the determination process of the access frequency comprises:

[0179] The access times of the strips are extracted based on a preset time window sliding.

[0180] The access frequency is determined by ratio operation of the access times and the window length of the time window.

[0181] Optionally, the hotness parameter updating module 502 is specifically configured to:

[0182] For the strip whose strip hotness level of the current configuration version number is inconsistent with the associated current strip hotness level, the corresponding current strip hotness level is matched with the strip configuration parameter threshold value.

[0183] The local repairable code strip configuration parameters of the corresponding strip are adjusted according to the strip configuration parameter threshold value, and the current configuration version number is updated.

[0184] Optionally, the scheduling cluster determining module 504 is specifically configured to:

[0185] The clusters of the distributed storage system are traversed, and the scheduling priority of each cluster with respect to each change data block is calculated based on the bandwidth of the change data block to the cluster and whether the change data block and the belonging strip are located in the same cluster.

[0186] The cluster with the maximum scheduling priority of each change data block is taken as the corresponding priority scheduling cluster.

[0187] Optionally, the data writing updating module 506 is specifically configured to:

[0188] The target writing node of each priority scheduling cluster is determined according to the storage available capacity ratio of each storage node in each priority scheduling cluster and the available bandwidth of the writing path.

[0189] If the load ratio of the distributed storage system is not greater than the high load threshold value, the associated change data block and local parity check block are written to the corresponding priority scheduling cluster according to each target writing node, and the corresponding strip is updated.

[0190] Optionally, the load storage parameter updating module is further configured to:

[0191] The average storage availability ratio and the load ratio of the distributed storage system are detected in real time.

[0192] The adjustment strategy of the local repairable code strip configuration parameters of each strip is determined according to the average storage availability ratio and the load ratio.

[0193] The embodiment of the application further provides a computer device comprising a memory and a processor, and the memory stores a computer program; when the computer program is executed by the processor, the processor executes the steps of the distributed storage optimization method based on dynamic configuration according to any one of the above embodiments.

[0194] The embodiment of the present application further provides a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the dynamic configuration based distributed storage optimization method according to any one of the above embodiments.

[0195] The embodiment of the present application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the dynamic configuration based distributed storage optimization method according to any one of the above embodiments.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and module can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner for actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0198] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0199] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0200] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0201] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dynamic configuration based distributed storage optimization method, characterized in that, The method is applied to a distributed storage system, and comprises the following steps: Real-time collection of access frequencies of each strip in the distributed storage system, and determination of current strip heat levels of each strip based on the access frequencies; For a strip whose current configuration version number of strip heat level is inconsistent with the associated current strip heat level, updating the current configuration version number and the local repairable code strip configuration parameter based on the corresponding current strip heat level; Real-time identification of incremental data in each strip to determine changed data blocks; Traversal of clusters of the distributed storage system to determine the priority scheduling clusters of each changed data block; Local recoding of each changed data block to update the local parity check block associated with each changed data block; Writing of each changed data block and the corresponding local parity check block into the corresponding priority scheduling cluster, and updating of the corresponding strip.

2. The method of claim 1, wherein, The determination process of the access frequency comprises the following steps: Sliding extraction of the access frequency of each strip based on a preset time window; Determination of the access frequency by ratio operation of the access frequency and the window length of the time window. 3.The method of claim 1, wherein, For a strip whose current configuration version number of strip heat level is inconsistent with the associated current strip heat level, updating the current configuration version number and the local repairable code strip configuration parameter based on the corresponding current strip heat level, which comprises the following steps: Matching of the strip configuration parameter threshold based on the corresponding current strip heat level for a strip whose current configuration version number of strip heat level is inconsistent with the associated current strip heat level; Adjustment of the local repairable code strip configuration parameter of the corresponding strip according to each strip configuration parameter threshold, and updating of the current configuration version number.

4. The method of claim 1, wherein, The traversal of clusters of the distributed storage system to determine the priority scheduling clusters of each changed data block comprises the following steps: Traversal of clusters of the distributed storage system to calculate the scheduling priority of each cluster with respect to each changed data block based on the bandwidth of the changed data block to the cluster and whether the changed data block and the strip belong to the same cluster; Taking the cluster with the maximum scheduling priority of each changed data block as the corresponding priority scheduling cluster.

5. The method of claim 1, wherein, The writing of each changed data block and the corresponding local parity check block into the corresponding priority scheduling cluster, and the updating of the corresponding strip, which comprises the following steps: Determination of the target write node of each priority scheduling cluster according to the storage available capacity ratio and the write path available bandwidth of each storage node in each priority scheduling cluster; If the load ratio of the distributed storage system is not greater than a high load threshold, writing of the associated changed data block and the local parity check block into the corresponding priority scheduling cluster according to each target write node, and updating of the corresponding strip.

6. The method of claim 1, wherein, Further comprising the following steps: Real-time detection of the average storage availability ratio and the load ratio of the distributed storage system; Determination of the adjustment strategy of the local repairable code strip configuration parameter of each strip according to the average storage availability ratio and the load ratio.

7. A dynamic configuration based distributed storage optimization apparatus, characterized in that, The device is applied to a distributed storage system, and comprises the following modules: An access behavior detection module for real-time collection of access frequencies of each strip in the distributed storage system, and determination of current strip heat levels of each strip based on the access frequencies; The heat parameter updating module is configured to update the current configuration version number and the local repairable code strip configuration parameter based on the corresponding current strip heat level for the strip whose current configuration version number is inconsistent with the associated current strip heat level. The incremental data identification module is configured to identify incremental data in each of the strips in real time and determine changed data blocks. The scheduling cluster determination module is configured to traverse the clusters of the distributed storage system and determine the priority scheduling clusters of each of the changed data blocks. The check block re-encoding module is configured to perform local re-encoding on each of the changed data blocks and update the local parity check blocks associated with each of the changed data blocks. The data writing updating module is configured to write each of the changed data blocks and the corresponding local parity check blocks to the corresponding priority scheduling cluster and update the corresponding strip.

8. A computer device, comprising: The computer program / instruction is executed by the processor to implement the steps of the distributed storage optimization method based on dynamic configuration according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the distributed storage optimization method based on dynamic configuration according to any one of claims 1-6.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the distributed storage optimization method based on dynamic configuration according to any one of claims 1-6.

Images