High-speed distributed storage system and method

Abstract

The invention relates to the technical field of distributed storage, in particular to a high-speed distributed storage system and method, and the system comprises a file index characterization module which carries out the statistics of the access frequency and the recent access timestamp of each data block in the system, and obtains the popularity magnitude of the file data blocks; according to the method, the popularity magnitude concept is introduced on the basis of linkage statistics of the data block access frequency and the recent access time, the data blocks with remarkable service influence in the current system can be recognized, the priority processing sequence of different data blocks in the repair process is divided, and the decision-making efficiency during fault recovery is improved. After the to-be-repaired data blocks are identified, the comprehensive weight values are formed through correlation calculation with the popularity magnitude, and all the weight values are arranged in a descending order, so that dynamic grading and sequential scheduling of repair tasks are realized, and important data blocks are guaranteed to preferentially obtain recovery resource support.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a high-speed distributed storage system and method. Background Technology

[0002] Distributed storage refers to an architecture that distributes data across multiple physical server nodes to create a unified logical storage resource pool. It is a key foundational technology in computer storage systems.

[0003] Current technologies rely solely on the number of data replicas, physical node balance, or static priority as references for recovery scheduling, failing to adequately identify the correlation between data blocks and business scenarios. This leads to resource allocation imbalances in high-concurrency repair scenarios, impacting the timely recovery of critical data. Furthermore, the lack of dynamic sampling and response mechanisms for real-time link bandwidth status and disk read / write load during path selection and data scheduling can result in forced transmission operations even when paths are congested or device loads are excessive, causing link backlog, increased retransmission rates, IO resource contention, and even affecting the read / write performance of normal business operations. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and to propose a high-speed distributed storage system and method.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a high-speed distributed storage system comprising: The file indexing feature module statistically analyzes the access frequency and most recent access timestamp of each data block in the system to obtain the file data block heat level. The differential repair sequence characterization module associates the file data block heat level corresponding to each lost data block with the list of data blocks lost after node failure and the heat level of the file data block, obtains the comprehensive weight value of the data block to be repaired, and then sorts all the comprehensive weight values ​​of the data blocks to be repaired in descending order to establish a hierarchical repair task queue. The repair path traffic scheduling module reads the first task in the hierarchical repair task queue, retrieves the I / O path between the source storage node and the target storage node required by the task, collects the network bandwidth utilization and disk read / write queue length of the I / O path, generates the target I / O path load, compares the target I / O path load with a preset load threshold, and sets the upper limit of the data transmission rate for repair after passing the test, thus forming a data reconstruction instruction set. The data replica reconstruction execution module receives the data reconstruction instruction set, performs data reconstruction on the target storage node according to the set data transmission rate limit, generates a reconstructed data block verification sequence, and submits the location information and status change request of the new data block to the metadata node after the reconstructed data block verification sequence is consistent, and obtains the updated file system metadata table.

[0006] Preferably, the step of obtaining the heat level of the file data block is as follows: Based on the operation logs of the distributed storage system, the access records of all data blocks are traversed, the cumulative number of accesses for each data block within the target monitoring period and the last access timestamp are extracted, and the business identification dictionary in the file system is called to extract the business key flag value bound to the data block, generating the total number of accesses, the last access timestamp and the business key flag value for each data block. The popularity value of each data block is calculated based on the total number of accesses, the last access timestamp, and the business criticality indicator value. Based on the popularity value of each data block, all data blocks are mapped according to their popularity values ​​and assigned to a popularity field under a unified dimension, generating a file data block popularity level.

[0007] Preferably, the step of obtaining the comprehensive weight value of the data block to be repaired is as follows: Based on the identifier of the faulty node in the node fault record table, extract the block number list of all lost data blocks on the corresponding node, and retrieve the heat value of each lost data block in combination with the file data block heat level. At the same time, aggregate by node dimension to form the sum of heat values ​​of all lost data blocks in each faulty node, and obtain the sum of file data block heat value of each lost data block and the node heat loss of the node to which it belongs. Calculate the comprehensive weight value of the data block to be repaired based on the sum of the file data block heat value of each lost data block and the node heat loss of its respective node.

[0008] Preferably, the step of obtaining the hierarchical repair task queue is as follows: Based on the comprehensive weight value of the data block to be repaired, all data blocks to be repaired are traversed, the corresponding comprehensive weight value is extracted, and the key-value pair of the block number and comprehensive weight value of each data block to be repaired is recorded to generate a mapping table between the data block number and the comprehensive weight value. Based on the mapping table between the data block number to be repaired and the comprehensive weight value, the data block numbers are sorted from largest to smallest according to the comprehensive weight value field, and the sorted data block numbers are rewritten into the queue array in order to generate a sorted priority array of data blocks to be repaired. Based on the sorted priority array of data blocks to be repaired, they are pushed into the scheduling pool in sequence according to the queue data structure format. The head of the queue is set as the primary processing object, and the repair plan is arranged in order from the head to the tail of the queue to generate a hierarchical repair task queue.

[0009] Preferably, the step of obtaining the target I / O path load is as follows: Based on the hierarchical repair task queue, extract the number of the data block to be repaired recorded at the current head of the queue, obtain the storage location information of the corresponding data block, record the source storage node identifier and target storage node identifier of the data block to be repaired, and generate the node information record of the task to be repaired at the head of the queue. Based on the node information record of the task to be repaired at the head of the queue, traverse the set of path nodes between the source storage node and the target storage node, record the path node sequence and the network link number between adjacent path nodes, and generate a complete I / O path mapping set between the source storage node and the target storage node. Based on the complete I / O path mapping set, the current network bandwidth utilization rate corresponding to the link number and the disk read / write queue length recorded by the disk management system of the path node are extracted one by one. The network bandwidth utilization rate and the disk read / write queue length are normalized and then summed to generate the target I / O path load.

[0010] Preferably, the step of obtaining the data reconstruction instruction set is as follows: Based on the target I / O path load, the difference between the target I / O path load and the preset load threshold is calculated for each path. The result of the difference calculation is used to determine whether the target I / O path load is in the state where data transmission is allowed, and a comparison and determination result of the target I / O path load is generated. Based on the comparison and judgment results of the target I / O path load, a set of path nodes whose difference calculation results meet the allowable data transmission state of the path load are selected, a predefined data transmission rate upper limit configuration item is obtained, the data transmission rate upper limit of this repair task is set for each node, and a data reconstruction instruction set is generated.

[0011] Preferably, the step of obtaining the reconstructed data block verification sequence is as follows: Based on the data reconstruction instruction set, the block number, source storage node identifier, target storage node identifier, and set data transmission rate limit of the data blocks to be repaired are extracted one by one to establish a data reconstruction task scheduling list and generate a complete data reconstruction task information record. Based on the complete data reconstruction task information record, read requests are sent to the source storage nodes one by one. The data transmission flow is controlled according to the data transmission rate limit parameter. The data copy or erasure code block corresponding to the data block to be repaired is read into the temporary data cache area, and the data cache copy read by the source storage node is generated. Based on the data cache copy read from the source storage node, data blocks are reconstructed one by one to the target storage node. The cache copy is restored and the data integrity is compared block by block. The hash verification value of the reconstructed data block is extracted and sorted according to the reconstruction task order to generate a reconstructed data block verification sequence.

[0012] Preferably, the step of obtaining the updated file system metadata table is as follows: Based on the reconstructed data block verification sequence, the hash verification value in the reconstructed data block verification sequence is compared with the corresponding original data block hash verification value record one by one to determine whether the hash verification values ​​of the reconstructed data block and the original data block match, and a reconstructed data block comparison result record is generated. Based on the reconstructed data block comparison results, filter the node information corresponding to the data blocks with matching hash check values, extract the physical storage location information of each matching data block in the target storage node and the status identifier of the data block reconstruction success, and generate a new data block location and status update request. Based on the new data block location and status update requests, the interface provided by the file system metadata node is called one by one, and the physical storage location information and status identifier data in each request are submitted. The update confirmation message returned by the metadata node is monitored in real time, and the update completion identifier in the confirmation message is recorded one by one into the file system metadata table to generate an updated file system metadata table.

[0013] This invention also provides a high-speed distributed storage method, comprising the following steps: The access frequency and most recent access timestamp of each data block in the system are statistically analyzed to obtain the file data block heat level; Based on the list of data blocks lost after node failure and the heat level of the file data blocks, the heat level of the file data blocks corresponding to each lost data block is associated to obtain the comprehensive weight value of the data blocks to be repaired. Then, all the comprehensive weight values ​​of the data blocks to be repaired are sorted in descending order to establish a hierarchical repair task queue. Read the first task in the hierarchical repair task queue, retrieve the I / O path between the source storage node and the target storage node required by the task, collect the network bandwidth utilization and disk read / write queue length of the I / O path, generate the target I / O path load, compare the target I / O path load with the preset load threshold, and set the upper limit of the data transmission rate for repair after passing the test, thus forming a data reconstruction instruction set. Upon receiving the data reconstruction instruction set, and based on the set upper limit of data transmission rate, perform data reconstruction on the target storage node, generate a reconstructed data block verification sequence, and submit the location information and status change request of the new data block to the metadata node after the reconstructed data block verification sequence is consistent, and obtain the updated file system metadata table.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention introduces the concept of heat level based on the linked statistics of data block access frequency and recent access time. This allows for the identification of data blocks with significant business impact in the current system, prioritizing different data blocks during the repair process, and improving decision-making efficiency during fault recovery. After identifying the data blocks to be repaired, a comprehensive weight value is calculated based on the correlation with the heat level. All weight values ​​are then sorted in descending order, enabling dynamic hierarchical and sequential scheduling of repair tasks, ensuring that important data blocks receive priority access to recovery resources. For data transmission paths, a dynamic monitoring mechanism for bandwidth utilization and disk read / write queue length is introduced. Before data transmission, the path load is assessed in real time, and a data transmission rate limit is set accordingly to avoid read / write blocking or system jitter due to path overload. During the data transmission phase, data copies or erasure blocks are stably extracted from the source storage node through limited rate control, and data reconstruction and hash checksum comparison operations are completed at the target storage node to ensure the reliability of the reconstructed data in terms of consistency and integrity. After completing the consistency check, update requests are uniformly submitted to the metadata node to ensure that the metadata table remains synchronized with the physical data state. It enhances the autonomous recovery capability in node failure scenarios, the closed-loop security of the data processing chain, and the response efficiency of the resource scheduling process. Attached Figure Description

[0015] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] Please see Figure 1 This invention provides a technical solution: a high-speed distributed storage system comprising: The file indexing feature module statistically analyzes the access frequency and most recent access timestamp of each data block in the system to obtain the file data block heat level. The differential repair sequence characterization module associates the file data block heat level corresponding to each lost data block with the list of lost data blocks and the heat level of file data blocks after node failure to obtain the comprehensive weight value of the data blocks to be repaired. Then, it sorts all the comprehensive weight values ​​of the data blocks to be repaired in descending order to establish a hierarchical repair task queue. The repair path traffic scheduling module reads the first task in the hierarchical repair task queue, retrieves the I / O path between the source storage node and the target storage node required by the task, collects the network bandwidth utilization and disk read / write queue length of the I / O path, generates the target I / O path load, compares the target I / O path load with the preset load threshold, and sets the upper limit of the data transmission rate for repair after passing the test, thus forming a data reconstruction instruction set. The data replica reconstruction execution module receives the data reconstruction instruction set, performs data reconstruction on the target storage node according to the set data transmission rate limit, generates a reconstructed data block verification sequence, and submits the location information and status change request of the new data block to the metadata node after the reconstructed data block verification sequence is consistent, and obtains the updated file system metadata table.

[0018] The steps to obtain the heat level of file data blocks are as follows: Based on the operation logs of the distributed storage system, the access records of all data blocks are traversed, the cumulative number of accesses for each data block within the target monitoring period and the last access timestamp are extracted, and the business identification dictionary in the file system is called to extract the business key flag value bound to the data block, generating the total number of accesses, the last access timestamp and the business key flag value for each data block. The popularity score of each data block is calculated based on its total number of accesses, last access timestamp, and business criticality indicator value. The calculation formula is as follows: ; in, Indicates the first The heat value of each data block Indicates the first The total number of accesses to each data block within the monitoring period. Indicates the first The last access timestamp of each data block Indicates the current system timestamp. Indicates the first Standardized business-critical flag values ​​associated with each data block. Indicates the first Business impact factors calculated from each data block This represents a constant used to adjust the degree of business criticality. This represents the time decay coefficient used to measure the rate of heat decay; Based on the popularity value of each data block, all data blocks are mapped according to their popularity values ​​and assigned to a popularity field under a unified dimension, generating a file data block popularity level.

[0019] Specifically, based on the operation logs of the distributed storage system, which record detailed access activities of all data blocks within the system in a structured format, each log entry typically includes an access timestamp, a unique identifier of the accessed data block, the access type (e.g., read, write), and the user ID or application ID that initiated the access. First, a clear target monitoring period is set, for example, defined as the past 24 hours. This period is configured by the system administrator based on dynamic changes in system load characteristics and data access patterns to ensure the timeliness of heat analysis. Next, the system iterates through all log entries recorded within this 24-hour period. For each unique data block ID, the total number of read and write operations is accumulated to obtain the cumulative access count for that data block within the target monitoring period. Simultaneously, for each data block, the record with the most recent timestamp is selected from all its access records and extracted as its last access timestamp. At the same time, the system calls pre-... First, a file system business identifier dictionary is established and maintained. This dictionary stores the mapping relationship between data block IDs and their corresponding original business attributes (e.g., "core transaction data", "user personal information", "application configuration files", "historical archived logs"). This dictionary is updated regularly by the data governance team based on the results of Business Impact Analysis (BIA). For each data block's original business attribute found in the dictionary, it is converted into a numerical business criticality indicator value according to a set of predefined quantification rules. For example, "core transaction data" is set to 5 points, "user personal information" to 4 points, "application configuration files" to 3 points, "general application data" to 2 points, and "historical archived logs" to 1 point. This quantification rule ensures that different business importance levels can be uniformly measured. Finally, a triplet information record containing the total number of accesses, the last access timestamp (in Unix timestamp format), and the quantified business criticality indicator value is generated for each data block in the system.

[0020] formula: , The advantage of this formula is that it takes into account the frequency of data block access. ), access proximity ( ) and business criticality ( ), through logarithmic function To smooth out the extreme impact of access frequency, this avoids the excessive dominance of the overall sorting by the popularity value of extremely high-frequency access data blocks, ensuring that medium- and low-frequency access data blocks also have the opportunity to obtain a reasonable popularity assessment based on their relevance and business importance, with an exponential decay term. This allows data block popularity to naturally decline over time, more accurately reflecting current data access hotspots and business impact factors. As an index, it can amplify or adjust the impact of business criticality on the final popularity value, so that critical business data can maintain a high popularity even if it is not accessed most frequently or recently, thus being prioritized in scenarios such as resource allocation, data migration, and fault recovery.

[0021] Indicates the first The total number of accesses to a data block within the monitoring period is obtained directly by parsing the total number of accesses for each data block. For example, within a set 24-hour monitoring period, analyzing the distributed storage system's operation logs and finding that data block_A has been accessed a total of 500 times, then... .

[0022] Indicates the first The last access timestamp of each data block is obtained directly from the last access timestamp of each data block generated during parsing. It is usually expressed as the number of seconds or milliseconds since a certain epoch (such as January 1, 1970, 00:00:00 UTC). For example, the last access timestamp of data block_A is recorded as 1678886400 (corresponding to a specific date and time).

[0023] This represents the current system timestamp, i.e., the system time when the popularity calculation was performed, and is similar to... The same time unit and format are used for implementation, for example, if the current calculation time is 1678890000.

[0024] Indicates the first The standardized business criticality flag value associated with each data block is derived from the generated business criticality flag value and standardized to ensure that its value falls within a uniform range (for...). This facilitates comparison of data blocks from different business types, and the standardized calculation formula can adopt the min-max normalization method: ,in It is the original business criticality score. and These are the minimum and maximum values ​​in the original scoring system. For example, if the original score is 1 to 5, for a data block block_A with a score of 5 (core transaction data), its standardized business key indicator value is... For a data block with a score of 3 (application configuration file), its .

[0025] This represents a constant used to adjust the degree of impact of business criticality. The setting of this constant is based on the system's sensitivity requirements for business criticality; the larger the value, the more significant the effect of business criticality on increasing the popularity value. The determination is usually based on empirical tuning or A / B testing. For example, an initial value (such as 0.5) can be set, and then different values ​​can be observed. Consider the impact of the value on the heat distribution and subsequent factors (such as repair priority), and select the one that best meets business needs. Value, implementation, for example, after testing, it was found that when If the popularity of high-business-value data blocks can be effectively improved and the sorting meets expectations, then set... .

[0026] This represents the time decay coefficient used to measure the rate at which data block popularity decays. This coefficient determines how quickly the popularity of a data block decreases over time. It can be set based on the typical half-life of data access, i.e., how long it takes for the near-time weight in the data block popularity to decay by half. The calculation formula is as follows: ,in This is the set half-life of access frequency. For example, if you want the access frequency of a data block to be within 7 days (…), then… After 1 hour, its short-term effect decays by half. .

[0027] Calculation process: Taking data block_A as an example, substitute specific values ​​to perform the calculation: Total number of visits during the monitoring period Second-rate.

[0028] Last access timestamp Second.

[0029] Current system timestamp Second.

[0030] Time difference Seconds (i.e., 1 hour).

[0031] Standardized Business Key Indicator Values .

[0032] Business-critical constants .

[0033] .

[0034] First, calculate the business impact factors. : ; Then calculate the data block heat value. , Hour, : ; ; ; ; ; ; ; The result indicates that the heat value of data block_A is calculated to be approximately 38.216. This value is a relative metric used to compare the heat values ​​of other data blocks. The higher the value, the "hotter" the data block is under the current comprehensive evaluation system (combining access frequency, proximity, and business importance). In subsequent storage strategy decisions, such as data tiering, cache management, or fault recovery priority ranking, data blocks with higher heat values ​​usually receive higher priority.

[0035] Based on the heat value of each data block calculated in the previous step, for example, the heat value of data block_A is 38.216, the heat value of data block_B is 15.0, etc., a set containing all data blocks and their corresponding original heat values ​​is formed. In order to eliminate the huge differences that may exist between the heat values ​​of different data blocks and make them comparable, these heat values ​​need to be mapped and assigned to a heat field under a unified unit. The specific operation is as follows: first, traverse the heat values ​​of all data blocks in the set and find the maximum heat value ( ) and minimum heat value ( Then, the raw heat value for each data block. The min-max normalization method is applied for processing, and the calculation formula is as follows: ,in This is the upper limit of the selected uniform dimension; for example, a range of 0-100 can be selected. If the calculated heat values ​​of all data blocks are obtained, , So for block_A (original heat value 38.216), its normalized heat value is... For block_B (original heat value 15.0), its normalized heat value is The original popularity values ​​of all data blocks are converted into a standardized popularity score between 0 and 100 in this way. These standardized popularity scores constitute the popularity scale of the file data blocks. This scale is represented as a mapping table or list, where each entry contains a unique identifier of the data block and its corresponding standardized popularity field value.

[0036] The steps to obtain the comprehensive weight value of the data block to be repaired are as follows: Based on the identifier of the faulty node in the node fault record table, extract the block number list of all lost data blocks on the corresponding node, and retrieve the heat value of each lost data block by combining the file data block heat level. At the same time, aggregate by node dimension to form the sum of heat values ​​of all lost data blocks in each faulty node, and obtain the sum of file data block heat value of each lost data block and the node heat loss of the node to which it belongs. Based on the sum of the file data block heat value and the node heat loss of each lost data block, the comprehensive weight value of the data block to be repaired is calculated. The calculation formula is as follows: ; in, For the first The overall weight value of each data block to be repaired. For the first The file data block heat value of the data block to be repaired. All of the tasks in this repair mission The sum of the heat values ​​of each lost data block Indicates the first The faulty node to which each data block belongs. To be with data blocks The sum of the heat values ​​of all lost data blocks located at the same fault node. This is a risk amplification factor used to amplify the impact of node risk concentration.

[0037] Specifically, based on the fault node identifiers in the node fault log table maintained by the system, which records the information of storage nodes that have failed in real time or near real time, each record contains at least a unique network identifier (e.g., IP address or hostname) of the fault node and a timestamp of the failure. The system first extracts the identifiers of one or more fault nodes that need to be processed from this table. For each identified fault node, the system then queries the metadata service or internal state table of the distributed file system to obtain a list of unique block numbers of all data blocks stored on that fault node. These data blocks are temporarily inaccessible due to the node failure and are considered lost data blocks. After forming an initial list of lost data blocks, the system combines the previously calculated and generated "file data block heat level" (a mapping structure containing data block IDs and their corresponding 0-100 normalized heat values) to encode each lost data block in the list. The system retrieves the heat values ​​of the corresponding file data blocks by number. For example, if the list of lost data blocks includes block_X and block_Y, their heat values ​​are retrieved from the "File Data Block Heat Level". For instance, the heat value of block_X is 75.3 and the heat value of block_Y is 42.1. After obtaining the heat values ​​of all lost data blocks, the system further performs aggregation operations according to the dimension of the fault node. Specifically, for each fault node, the file data block heat values ​​of all lost data blocks on it are summed to calculate the total heat value of all lost data blocks within that fault node. This total represents the "heat loss" caused by the fault of that specific node. Finally, each lost data block is associated with two key values: the file data block heat value of the data block itself, and the total heat loss of the fault node to which it belongs.

[0038] formula: The advantage of this formula is that it can more intelligently determine the repair priority of data blocks, and it introduces the concentration of hot data loss on the faulty node as an influencing factor. This ensures that blocks with higher popularity inherently have a higher base repair weight, while This further amplifies the repair weight of data blocks located on "severely affected" nodes (i.e., nodes that have lost a large amount of total heat). This design enables the system to prioritize the repair of data that has a greater impact on business and is more risky when facing multiple concurrent failures or large-scale data loss. For example, if a node loses a large amount of critical business data (with a large overall heat loss), even if the individual heat of a certain data block is slightly lower than the hottest data block on another isolated failed node.

[0039] Indicates the first The file data block heat value for the data block to be repaired is directly derived from the standardized heat value (e.g., a range of 0-100) obtained in the previous step by querying the "file data block heat level" for each lost data block. This value comprehensively reflects the access frequency, relevance, and business criticality of the data block. For example, for the data block to be repaired, block_X, its file data block heat value determined in the previous step is 75.3, then... .

[0040] This indicates that all of the tasks in this repair mission The sum of the heat values ​​of each lost data block is calculated after all data blocks lost due to node failure have been identified, by assigning their respective heat values ​​to each data block. The sum of the file data block popularity values ​​represents the overall popularity loss caused by this failure event. This is achieved by iterating through all identified lost data blocks and summing their individual popularity values. Value. For example, if this fault involves three lost data blocks with heat values ​​of 75.3, 42.1, and 90.5 respectively, then... .

[0041] Indicates the first The faulty node to which a data block belongs is a logical attribution relationship, indicating the data block's ownership. The information about which physical or logical node the data was stored on before the failure was known when retrieving the list of lost data blocks. For example, the failed node to which data block_X belonged was Node_Alpha.

[0042] Representation and data block The sum of the heat values ​​of all lost data blocks located at the same faulty node, i.e., the "sum of node heat loss of the associated node" calculated in the previous step, quantifies the data block. The total amount of heat lost on that specific faulty node. For example, if the faulty node Node_Alpha lost block_X (heat 75.3) and block_Y (heat 42.1), then for block_X, .

[0043] This represents a risk amplification factor used to amplify the impact of node risk concentration. This factor is designed to more accurately reflect the additional weighting of repair priority due to the concentrated risk of losing a large amount of high-priority data caused by a single node failure. It is determined based on analysis of the system's overall risk tolerance, the distribution of business criticality, and historical failure impact data. For example, based on historical data analysis, the relationship between the concentration of high-priority data loss in node failures and the recovery time to order (RTO) or data loss impact (RPO) is analyzed. If it is found that the proportion of high-priority data loss in a node failure (i.e.,...) is high... If the impact on business grows exponentially when a certain threshold is exceeded (e.g., 0.3), a relatively high threshold will be set. Values ​​(implemented as 1.0 to 1.5) are used to prioritize responses to these high-risk, concentrated failures; conversely, if the business is not sensitive to data distribution, then... The value can be set relatively low (e.g., 0.2 to 0.5) to avoid excessively amplifying the node concentration effect. In this embodiment, through stress testing and fault simulation of the system, it was found that when the node heat loss accounts for more than 40% of the total loss, the performance degradation of critical applications is particularly significant. To balance the heat of individual data blocks with the overall risk of nodes, a value is set... This value is chosen to allow the node concentration factor to increase its weight by approximately 30% to 50% under high-risk conditions (when...). (When the size is large), thus effectively adjusting the repair sequence.

[0044] Calculation process: A system failure caused two storage nodes, Node_A and Node_B, to fail.

[0045] Data block B1 is missing on Node_A; its file data block heat value is... .

[0046] Data block B2 is missing on Node_A; its file data block heat value is... .

[0047] Therefore, the total heat loss of Node_A .

[0048] Data block B3 is missing on Node_B; its file data block heat value is... .

[0049] Therefore, the total node heat loss of Node_B .

[0050] Total number of lost data blocks in this repair task .

[0051] Sum of the heat values ​​of all lost data blocks .

[0052] Risk amplification factor .

[0053] Calculate the overall weight value of data block B1 : ; ; ; ; ; ; ; Calculate the overall weight value of data block B2 : ; ; ; ; ; ; ; Calculate the overall weight value of data block B3 : ; ; ; ; ; ; ; The results show that the comprehensive weight value of data block B1 is 0.4864, the comprehensive weight value of data block B2 is 0.4256, and the comprehensive weight value of data block B3 is 0.5081. Data blocks with larger values ​​have higher repair priority. In this example, although B3 (heat level 95) has a higher individual heat level than B1 (heat level 80), the total heat loss of Node_A (150) where B1 and B2 reside is much greater than the total heat loss of Node_B (95). The coefficient amplifies this difference, resulting in the final calculated value. (0.5081) is the highest, followed by (0.4864), and finally... (0.4256), these weight values ​​will be used directly to build the hierarchical repair task queue.

[0054] The steps to obtain the tiered repair task queue are as follows: Based on the comprehensive weight value of the data block to be repaired, all data blocks to be repaired are traversed, the corresponding comprehensive weight value is extracted, and the key-value pair of the block number and comprehensive weight value of each data block to be repaired is recorded to generate a mapping table between the data block number and the comprehensive weight value. Based on the mapping table between the data block number to be repaired and the comprehensive weight value, the data blocks are sorted from largest to smallest according to the comprehensive weight value field. The sorted data block numbers are then rewritten into the queue array in order to generate a sorted priority array of data blocks to be repaired. Based on the sorted priority array of data blocks to be repaired, they are pushed into the scheduling pool in sequence according to the queue data structure format. The head of the queue is set as the primary processing object, and the repair plans are arranged in order from the head to the tail of the queue to generate a hierarchical repair task queue.

[0055] Specifically, based on the comprehensive weight values ​​of each data block to be repaired calculated in the previous step (for example, the comprehensive weight value of data block B1 is 0.4864, the comprehensive weight value of data block B2 is 0.4256, and the comprehensive weight value of data block B3 is 0.5081), the system will initiate an iterative process, traversing the set of all data blocks currently marked as "to be repaired". For each data block in the set, the system reads its unique block number (for example, block_001, block_002, etc.) and the previously calculated and associated comprehensive weight value. Subsequently, the system organizes this information into a series of key-value pairs. (ePair), where the key is the block number of the data block and the value is the comprehensive weight value corresponding to the data block. For example, it forms a data structure such as {'block_001': 0.5081, 'block_002': 0.4864, 'block_003': 0.4256}. These key-value pairs are collected by the system and stored in a temporary memory structure, such as a hash table or dictionary. This process continues until all the data blocks to be repaired and their comprehensive weight values ​​have been successfully extracted and recorded, and finally a complete mapping table of data block numbers to be repaired and comprehensive weight values ​​is generated.

[0056] Based on the generated mapping table between the block numbers to be repaired and their comprehensive weight values, such as entries like {'block_001': 0.5081, 'block_002': 0.4864, 'block_003': 0.4256}, the system then performs a sorting operation. This sorting operation prioritizes each data block according to its comprehensive weight value, following a descending order. That is, data blocks with higher comprehensive weight values ​​appear earlier, indicating higher repair priority. The system will call sorting algorithms from the standard library, such as quicksort or mergesort, using the comprehensive weight value field as the sorting key to process all entries in the mapping table. During the sorting process, if data blocks with the same comprehensive weight value are encountered, a sorting algorithm can be used to resolve the issue. The system uses preset secondary sorting rules, such as sorting data blocks in ascending order by lexicographical order or in descending order by their original heat values, to ensure the uniqueness and stability of the sorting results. After sorting, the system will obtain an ordered list of elements with data block numbers, where the order of the data blocks is strictly arranged in descending order of their comprehensive weight values. For example, if the original mapping table is sorted and the order becomes block_001 (weight 0.5081), then block_002 (weight 0.4864), and finally block_003 (weight 0.4256), the system will write these sorted data block numbers into a new linear data structure, usually an array or list, in this order, thereby generating a sorted priority array of data blocks to be repaired.

[0057] Based on the generated sorted priority array of data blocks to be repaired, for example, the array content is ['block_001', 'block_002', 'block_003'], where block_001 has the highest repair priority. The system pushes the data block numbers in this priority array into a dedicated scheduling pool for managing repair tasks, according to their order in the array. This scheduling pool is usually implemented using a queue data structure, following the first-in, first-out (FIFO) principle. However, in this scenario, they are enqueued according to priority, effectively creating a priority queue effect. Specifically, the priority blocks are pushed into the queue. The first element in the hierarchical array (i.e., the data block number with the highest overall weight, such as block_001) is pushed into the scheduling pool as the head element. Then, the second element in the array (such as block_002) is pushed in, and so on, until all the data block numbers to be repaired in the array have entered the scheduling pool. The system explicitly sets the data block at the head of the scheduling pool as the current priority for repair processing. Subsequent data blocks are arranged in order of their position in the queue, forming the execution order of the repair plan. This structure ensures that repair resources are always preferentially allocated to the data block with the highest overall weight, that is, the data block that needs to be repaired the most at present, and finally a hierarchical repair task queue is generated.

[0058] The steps to obtain the target I / O path load are as follows: Based on the hierarchical repair task queue, extract the number of the data block to be repaired recorded at the current head of the queue, obtain the storage location information of the corresponding data block, record the source storage node identifier and target storage node identifier of the data block to be repaired, and generate the node information record of the task to be repaired at the head of the queue. Based on the node information record of the task to be repaired at the head of the queue, traverse the set of path nodes between the source storage node and the target storage node, record the path node sequence and the network link number between adjacent path nodes, and generate a complete I / O path mapping set between the source storage node and the target storage node. Based on the complete I / O path mapping set, the current network bandwidth utilization rate corresponding to the link number and the disk read / write queue length recorded by the disk management system of the path node are extracted one by one. The network bandwidth utilization rate and the disk read / write queue length are normalized and then summed to generate the target I / O path load.

[0059] Specifically, based on the hierarchical repair task queue generated in the previous step, for example, with the queue head being the data block number block_001, the system first performs a dequeue operation to extract the data block number to be repaired currently recorded at the queue head, i.e., block_001. Next, the system uses this data block number to query the file system's metadata service. This service maintains detailed distribution information for all data blocks, including the location of their replicas or the storage node information for erasure coding stripes. For the data block to be repaired, block_001, it is necessary to determine the storage nodes where its available source data replicas or erasure coding blocks reside; these nodes are called source storage nodes. Simultaneously, the system also needs to... Based on the preset data placement strategy or the current system resource status (such as node load, available space, etc.), one or more healthy storage nodes are selected as the target storage nodes for rebuilding data blocks. For example, a polling strategy is used to select the node with the lowest current load and sufficient space. Once the source storage node (e.g., SourceNode_X) and the target storage node (e.g., TargetNode_Y) are determined, the system records the unique identifiers (such as IP address or hostname) of these two nodes, and finally generates a node information record for the head of the queue of tasks to be repaired, which includes the number of the data block to be repaired, the identifier of the source storage node, and the identifier of the target storage node.

[0060] Based on the node information record of the task to be repaired at the head of the queue, which includes the source storage node identifier (e.g., SourceNode_X) and the target storage node identifier (e.g., TargetNode_Y), the system then initiates a network path discovery process. The purpose of this process is to determine all possible I / O data transfer paths between the source and target storage nodes. This process utilizes network topology information, maintained by the network management system or obtained through dynamic probing (e.g., traceroute tools). The system traverses all reachable intermediate network devices (e.g., switches, routers) and the storage nodes themselves between the source and target storage nodes, forming one or more path node sequences. For example, one path might be SourceNode_Y. For each identified path, ceNode_X->Switch_A->Router_1->Switch_B->TargetNode_Y, the system records the ordered sequence of nodes that make up the path in detail. Furthermore, for each pair of adjacent nodes in the path (e.g., SourceNode_X and Switch_A, Switch_A and Router_1, etc.), the system records the unique number or identifier of the network link used for data transmission between them. These link numbers correspond to specific physical or logical network connections. By summarizing all identified paths and their contained node sequences and link numbers, a complete I / O path mapping set between the source storage node and the target storage node is finally generated.

[0061] Based on the generated complete I / O path mapping set between the source storage node and the target storage node, for example, one of the paths is P1: SourceNode_X->Link_1->Switch_A->Link_2->Router_1->Link_3->Switch_B->Link_4->TargetNode_Y. The system will perform load assessment on each I / O path in this set. Specifically, it will extract the network link number (e.g., Link_1, Link_2, Link_3, Link_4) that constitutes the path, and obtain the current real-time network bandwidth utilization of these links by querying the network monitoring system (for example, using the SNMP protocol to obtain MIB data from the network device). The network bandwidth utilization is usually expressed as the ratio of used bandwidth to total bandwidth. The system calculates the network bandwidth utilization as a percentage. For example, Link_1 has a utilization rate of 60%. Simultaneously, the system queries the disk management systems of the storage nodes involved in the path (especially the source storage node SourceNode_X and the target storage node TargetNode_Y, as well as possible intermediate storage proxy nodes) to obtain their current disk read / write queue lengths. The disk read / write queue length reflects the level of disk I / O activity. For example, SourceNode_X has a disk read queue length of 10, and TargetNode_Y has a disk write queue length of 5. After obtaining the original network bandwidth utilization (percentage) and disk read / write queue length (number of requests), since these two metrics have different dimensions and numerical ranges, they need to be normalized separately, mapping them uniformly to, for example, a range of 0 to 1. The normalization formula can be Disk read / write queue length Normalization can be achieved using min-max normalization, for example... ,in It is usually 0. This could be a reasonable maximum queue length threshold set based on historical data or system design, for example, 100. If the current queue length is 10, then... The normalized bandwidth utilization of all links on the path is aggregated (implementation example: taking the average or maximum value), and then weighted and summed with the normalized disk read / write queue lengths of the nodes at both ends of the path (the weights can be set empirically, for example, network and disk each account for 0.5, or dynamically adjusted based on bottleneck analysis). The calculation formula can be: Path Load ,in and The weights are set to, for example, 0.5. In this way, a comprehensive load value is calculated for each I / O path. Finally, one or more paths with the lowest load values ​​are selected from all evaluated paths, and their load values ​​are the target I / O path load values.

[0062] The steps to obtain the data reconstruction instruction set are as follows: Based on the target I / O path load, the difference between the target I / O path load and the preset load threshold is calculated for each path. The result of the difference calculation determines whether the target I / O path load is in the state where data transmission is allowed, and a comparison and determination result of the target I / O path load is generated. Based on the comparison and judgment results of the target I / O path load, the set of path nodes whose difference calculation results meet the allowable data transmission state of the path load is selected, the predefined data transmission rate upper limit configuration item is obtained, the data transmission rate upper limit of this repair task is set for each node, and a data reconstruction instruction set is generated.

[0063] Specifically, based on the target I / O path load calculated in the previous step, for example, if the load of the selected optimal path P1 is 0.45, the system will compare this load value with a preset load threshold for each path. This preset load threshold is a key parameter set by the system administrator based on historical performance data, network capacity, and acceptable Quality of Service (QoS) requirements. For example, after long-term monitoring and analysis, if the normalized overall path load is below 0.7, the path is considered to be in good condition and data transmission is allowed. If the load is between 0.7 and 0.85, the path is considered busy but still allows transmission, but the rate needs to be limited. If it is above 0.85, the path is considered congested, and in principle, no new large-scale data transmission will be initiated. Here, a relatively lenient preset load threshold for allowing data transmission is set to 0.8. The system will calculate the difference between the target I / O path load and this preset load threshold, i.e. Based on this difference calculation result, the system makes a judgment. If the difference is greater than zero, that is, the load of the target I / O path is less than the preset load threshold, the path is determined to be in the state of allowing data transmission. If the difference is zero or less than zero, the path is determined to be unsuitable or requires caution in data transmission. For each target I / O path being evaluated, such a comparison judgment result will be generated, indicating whether it meets the load conditions for starting data transmission.

[0064] Based on the comparison results of the generated target I / O path load, for example, if the determination result for path P1 is "allowed for data transmission" (because its load of 0.45 is less than the preset load threshold of 0.8), the system will filter out all paths whose difference calculation results indicate that the path load is in the allowed data transmission state and their corresponding node sets. These nodes include the source storage node, the target storage node, and any intermediate forwarding nodes that may be involved in the path. For each available path selected, the system will then obtain a predefined data transmission rate upper limit configuration item. This configuration item is a policy table containing different transmission rate upper limits corresponding to different load intervals. This policy table is dynamically adjusted and set by the system administrator based on network bandwidth resources, storage node I / O capabilities, and the priority of different services (such as repair, normal read / write). For example, this configuration table may stipulate that when the path load is less than 0.5, the maximum allowed repair transmission rate is 50% of the theoretical link bandwidth. When the load is between 0.5 and 0.7, the maximum rate is 30% of the theoretical bandwidth of the link; when the load is between 0.7 and 0.8, the maximum rate is 10% of the theoretical bandwidth of the link. For path P1 (load 0.45), it falls into the "less than 0.5" range. If the theoretical bandwidth of the link is 1Gbps, its data transmission rate limit is set to 500Mbps. The system will set specific data transmission rate limits for each source storage node and target storage node involved in this repair task, based on the load of the selected path and the above configuration items. If there are multiple available paths, the system may select the optimal path or allocate traffic and set their respective rate limits according to the load balancing strategy. Finally, the information such as the data block number to be repaired, the source storage node, the target storage node, and the data transmission rate limits set for them are combined to form one or more specific data reconstruction instructions. All these instructions are then aggregated to generate a data reconstruction instruction set.

[0065] The steps for obtaining the reconstructed data block verification sequence are as follows: Based on the data reconstruction instruction set, the block number, source storage node identifier, target storage node identifier, and set data transmission rate limit of the data blocks to be repaired are extracted one by one to establish a data reconstruction task scheduling list and generate a complete data reconstruction task information record. Based on the complete data reconstruction task information record, read requests are sent to the source storage nodes one by one. The data transmission flow is controlled according to the data transmission rate limit parameter. The data copy or erasure code block corresponding to the data block to be repaired is read into the temporary data cache area, and the data cache copy read by the source storage node is generated. Based on the data cache copy read from the source storage node, data blocks are reconstructed one by one to the target storage node. The cache copy is restored and the data integrity is compared block by block. The hash check value of the reconstructed data block is extracted and sorted according to the reconstruction task order to generate a reconstructed data block check sequence.

[0066] Specifically, based on the data reconstruction instruction set generated in the previous step, for example, one instruction contains the data block number to be repaired as block_001, the source storage node identifier as SourceNode_X, the target storage node identifier as TargetNode_Y, and a set data transfer rate limit of 500Mbps. The system will parse these instructions one by one, accurately extracting the above four key information fields from each instruction: the unique block number of the data block to be repaired, the network identifier of the source storage node responsible for providing the original data copy or erasure code block, the network identifier of the target storage node used to store the reconstructed data block, and the maximum allowed rate that must be followed during data transfer. This extracted information is used to construct a... The data reconstruction task entries are structured and organized into a data reconstruction task scheduling list. This list details all data reconstruction operations to be executed and their related execution parameters, ensuring that each repair action has a clear execution basis and constraints. For example, a record in the list might be: {task_id: 1, block_id: 'block_001', source_node: 'SourceNode_X', target_node: 'TargetNode_Y', max_rate: '500Mbps'}. By summarizing these task entries formed after parsing all instructions, a complete data reconstruction task information record is finally generated.

[0067] Based on the generated complete data reconstruction task information record, for example, which includes tasks {task_id: 1, block_id: 'block_001', source_node: 'SourceNode_X', target_node: 'TargetNode_Y', max_rate: '500Mbps'}, the system will send data read requests to the specified source storage node (e.g., SourceNode_X) one by one according to the task order in the list. This request explicitly specifies the data block number to be read (e.g., block_001). After receiving the request, the source storage node will locate the corresponding data block replica or its associated erasure code block (if the system uses an erasure code mechanism) and begin reading the data. The data is transmitted over the network to the requesting repair coordinator or directly to the target storage node. During data transmission, the system strictly controls the data flow rate according to the upper limit parameter of the data transmission rate set for the task in the task information record (such as 500Mbps). This is usually implemented at the sending or receiving end through a flow control algorithm (such as the token bucket algorithm or the leaky bucket algorithm) to ensure that the repair process does not excessively occupy network bandwidth and affect normal business access. The read data block content is temporarily stored in a temporary memory data cache area. The size of this cache area needs to be reasonably configured according to the data block size and system resources. For example, if the data block size is 4MB, a cache space slightly larger than 4MB can be allocated. After successfully reading and caching the data, a cached copy of the data read by the source storage node is generated.

[0068] Based on the data cache copy read from the source storage node and generated, the data content of, for example, block_001, has been temporarily stored in the memory of the repair coordinator. Next, for each task in the complete data reconstruction task information record, the system will initiate a data block reconstruction operation to its specified target storage node (e.g., TargetNode_Y). This operation involves writing the data copy previously read from the source storage node into the temporary data cache to a new storage location on the target storage node. If erasure blocks are being read, this step requires reading enough erasure blocks from multiple source nodes and performing decoding operations on the target node or coordinator to restore the original data block content before writing it to the target storage node. After writing (or restoring and writing) to the target storage node, the system immediately performs data integrity verification on the newly generated data blocks. This is usually achieved by calculating the hash check value of the reconstructed data block (implemented by, for example, using SHA-256 or MD5 algorithms), and comparing the calculated hash check value with the hash check value of the original data block (if available). Alternatively, in erasure coding scenarios, the correctness of the check code can be used to indirectly confirm data integrity. For each successfully reconstructed and verified data block, the system records its newly generated hash check value and arranges them according to their original task order in the data reconstruction task information record to form an ordered list of hash values, ultimately generating a reconstructed data block verification sequence.

[0069] The steps for updating the file system metadata table are as follows: Based on the reconstructed data block verification sequence, the hash verification value in the reconstructed data block verification sequence is compared with the corresponding original data block hash verification value record one by one to determine whether the hash verification values ​​of the reconstructed data block and the original data block match, and a reconstructed data block comparison result record is generated. Based on the reconstruction data block comparison results, filter the node information corresponding to the data blocks with matching hash check values, extract the physical storage location information of each matching data block in the target storage node and the status identifier of the data block reconstruction success, and generate a new data block location and status update request. Based on the new data block location and status update request, the interface provided by the file system metadata node is called one by one, and the physical storage location information and status identifier data in each request are submitted. The update confirmation message returned by the metadata node is listened to in real time, and the update completion identifier in the confirmation message is recorded to the file system metadata table one by one to generate an updated file system metadata table.

[0070] Specifically, based on the reconstructed data block verification sequence generated in the previous step, such as including hash_A for block_001 and hash_B for block_002, the system will precisely compare the hash verification value calculated for each entry in the sequence, i.e., each reconstructed data block, with the corresponding hash verification value record of the original data block. The hash verification value record of the original data block is usually stored in the metadata of the file system or pre-calculated and saved when data is written. The system will query this record to obtain the hash value (e.g., original_ha) of a specific data block number (such as block_001) before it was damaged or its healthy copy. Then, the hash value (hash_A) of the reconstructed data block is compared bit by bit with the original hash value (original_hash_A) using strings or byte sequences. If the two are completely identical, the hash check value of the reconstructed data block is determined to match the hash check value of the original data block, indicating that the data reconstruction is successful and the data content has not been damaged. If the two are inconsistent, it is determined to be a mismatch, which may mean that an error occurred in the data reconstruction process. Such comparison judgment is performed for each reconstructed data block in the sequence, and the comparison result (e.g., "match" or "mismatch") along with the corresponding data block number is recorded, and finally a reconstructed data block comparison result record is generated.

[0071] Based on the generated comparison results of reconstructed data blocks, for example, if the record shows that the hash checksum of block_001 matches while that of block_002 does not, the system will first filter out the data block entries whose hash checksums successfully match. These entries represent data that has been successfully and correctly reconstructed on the target storage node. For each filtered matching data block, the system needs to obtain its specific physical storage location information on the target storage node. This information is usually returned by the target node's storage management service when the data is written to the target storage node, and may include detailed paths such as disk number, cylinder number, sector number, or higher-level logical block address. Simultaneously, the system will assign a "reconstruction successful" status flag to these successfully matched data blocks. (Example) If a boolean value is true or a specific status code is given, the following three information are combined: the number of each matching data block, its physical storage location on the target storage node, and the status identifier indicating that the data block has been successfully rebuilt. This combination forms a request to update the file system metadata. For example, for block_001, the request content could be: {block_id: 'block_001', new_location: 'TargetNode_Y: / disk3 / path / to / block_001', status: 'rebuilt_success'}. This operation is performed on all data blocks with matching hash check values, ultimately generating a series of new data block location and status update requests.

[0072] Based on the generated new data block location and status update requests, an implementation might include, for example, an update request for block_001: {block_id: 'block_001', new_location: 'TargetNode_Y: / disk3 / path / to / block_001', status: 'rebuilt_success'}. The system will process these requests one by one, calling the standard interface functions or APIs pre-provided by the file system metadata node (MDS), such as update_block_metadata(block_id, new_location, status). The physical storage location information (e.g., TargetNode_Y: / disk3 / path / to / block_001) and status identifier data (e.g., rebuilt_success) contained in each request are submitted to the metadata node as parameters. After receiving these update requests, the metadata node will search its internally maintained file system metadata table (usually a distributed database or key-value store) to find the corresponding data block. The system assigns an entry with a number (e.g., block_001) and updates its storage location information to the new physical address, while marking its status as available or indicating that it has been successfully repaired. After submitting the update request, the system listens in real time for update confirmation messages returned by the metadata node. This confirmation message usually contains a status code indicating whether the update operation was successful and the relevant transaction ID. For example, the metadata node may return {block_id: 'block_001', update_status: 'success', transaction_id: 'txn_123'}. After receiving a positive update confirmation message, the system records the identifiers indicating the update is complete (e.g., update_status: 'success') in the message one by one, usually in an operation log or a temporary status tracking table, to ensure that all successful metadata updates are properly recorded. This process continues until all new data block location and status update requests have been submitted and confirmed. Finally, it can be considered that an updated file system metadata table has been generated, even though the actual operation is to directly modify the existing metadata table rather than generate a completely new table.

Claims

1. A high-speed distributed storage system, characterized in that, The system includes: The file indexing feature module statistically analyzes the access frequency and most recent access timestamp of each data block in the system to obtain the file data block heat level. The differential repair sequence characterization module associates the file data block heat level corresponding to each lost data block with the list of data blocks lost after node failure and the heat level of the file data block, obtains the comprehensive weight value of the data block to be repaired, and then sorts all the comprehensive weight values ​​of the data blocks to be repaired in descending order to establish a hierarchical repair task queue. The repair path traffic scheduling module reads the first task in the hierarchical repair task queue, retrieves the I / O path between the source storage node and the target storage node required by the task, collects the network bandwidth utilization and disk read / write queue length of the I / O path, generates the target I / O path load, compares the target I / O path load with a preset load threshold, and sets the upper limit of the data transmission rate for repair after passing the test, thus forming a data reconstruction instruction set. The data replica reconstruction execution module receives the data reconstruction instruction set, performs data reconstruction on the target storage node according to the set data transmission rate limit, generates a reconstructed data block verification sequence, and submits the location information and status change request of the new data block to the metadata node after the reconstructed data block verification sequence is consistent, and obtains the updated file system metadata table.

2. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the heat level of the file data block are as follows: Based on the operation logs of the distributed storage system, the access records of all data blocks are traversed, the cumulative number of accesses for each data block within the target monitoring period and the last access timestamp are extracted, and the business identification dictionary in the file system is called to extract the business key flag value bound to the data block, generating the total number of accesses, the last access timestamp and the business key flag value for each data block. The popularity value of each data block is calculated based on the total number of accesses, the last access timestamp, and the business criticality indicator value. Based on the popularity value of each data block, all data blocks are mapped according to their popularity values ​​and assigned to a popularity field under a unified dimension, generating a file data block popularity level.

3. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the comprehensive weight value of the data block to be repaired are as follows: Based on the identifier of the faulty node in the node fault record table, extract the block number list of all lost data blocks on the corresponding node, and retrieve the heat value of each lost data block in combination with the file data block heat level. At the same time, aggregate by node dimension to form the sum of heat values ​​of all lost data blocks in each faulty node, and obtain the sum of file data block heat value of each lost data block and the node heat loss of the node to which it belongs. Calculate the comprehensive weight value of the data block to be repaired based on the sum of the file data block heat value of each lost data block and the node heat loss of its respective node.

4. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the tiered repair task queue are as follows: Based on the comprehensive weight value of the data block to be repaired, all data blocks to be repaired are traversed, the corresponding comprehensive weight value is extracted, and the key-value pair of the block number and comprehensive weight value of each data block to be repaired is recorded to generate a mapping table between the data block number and the comprehensive weight value. Based on the mapping table between the data block number to be repaired and the comprehensive weight value, the data block numbers are sorted from largest to smallest according to the comprehensive weight value field, and the sorted data block numbers are rewritten into the queue array in order to generate a sorted priority array of data blocks to be repaired. Based on the sorted priority array of data blocks to be repaired, they are pushed into the scheduling pool in sequence according to the queue data structure format. The head of the queue is set as the primary processing object, and the repair plan is arranged in order from the head to the tail of the queue to generate a hierarchical repair task queue.

5. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the target I / O path load are as follows: Based on the hierarchical repair task queue, extract the number of the data block to be repaired recorded at the current head of the queue, obtain the storage location information of the corresponding data block, record the source storage node identifier and target storage node identifier of the data block to be repaired, and generate the node information record of the task to be repaired at the head of the queue. Based on the node information record of the task to be repaired at the head of the queue, traverse the set of path nodes between the source storage node and the target storage node, record the path node sequence and the network link number between adjacent path nodes, and generate a complete I / O path mapping set between the source storage node and the target storage node. Based on the complete I / O path mapping set, the current network bandwidth utilization rate corresponding to the link number and the disk read / write queue length recorded by the disk management system of the path node are extracted one by one. The network bandwidth utilization rate and the disk read / write queue length are normalized and then summed to generate the target I / O path load.

6. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the data reconstruction instruction set are as follows: Based on the target I / O path load, the difference between the target I / O path load and the preset load threshold is calculated for each path. The result of the difference calculation is used to determine whether the target I / O path load is in the state where data transmission is allowed, and a comparison and determination result of the target I / O path load is generated. Based on the comparison and judgment results of the target I / O path load, a set of path nodes whose difference calculation results meet the allowable data transmission state of the path load are selected, a predefined data transmission rate upper limit configuration item is obtained, the data transmission rate upper limit of this repair task is set for each node, and a data reconstruction instruction set is generated.

7. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the reconstructed data block verification sequence are as follows: Based on the data reconstruction instruction set, the block number, source storage node identifier, target storage node identifier, and set data transmission rate limit of the data blocks to be repaired are extracted one by one to establish a data reconstruction task scheduling list and generate a complete data reconstruction task information record. Based on the complete data reconstruction task information record, read requests are sent to the source storage nodes one by one. The data transmission flow is controlled according to the data transmission rate limit parameter. The data copy or erasure code block corresponding to the data block to be repaired is read into the temporary data cache area, and the data cache copy read by the source storage node is generated. Based on the data cache copy read from the source storage node, data blocks are reconstructed one by one to the target storage node. The cache copy is restored and the data integrity is compared block by block. The hash verification value of the reconstructed data block is extracted and sorted according to the reconstruction task order to generate a reconstructed data block verification sequence.

8. The high-speed distributed storage system according to claim 1, characterized in that, The steps for obtaining the updated file system metadata table are as follows: Based on the reconstructed data block verification sequence, the hash verification value in the reconstructed data block verification sequence is compared with the corresponding original data block hash verification value record one by one to determine whether the hash verification values ​​of the reconstructed data block and the original data block match, and a reconstructed data block comparison result record is generated. Based on the reconstructed data block comparison results, filter the node information corresponding to the data blocks with matching hash check values, extract the physical storage location information of each matching data block in the target storage node and the status identifier of the data block reconstruction success, and generate a new data block location and status update request. Based on the new data block location and status update requests, the interface provided by the file system metadata node is called one by one, and the physical storage location information and status identifier data in each request are submitted. The update confirmation message returned by the metadata node is monitored in real time, and the update completion identifier in the confirmation message is recorded one by one into the file system metadata table to generate an updated file system metadata table.

9. The high-speed distributed storage method for a high-speed distributed storage system according to any one of claims 1-8, characterized in that, Includes the following steps: The access frequency and most recent access timestamp of each data block in the system are statistically analyzed to obtain the file data block heat level; Based on the list of data blocks lost after node failure and the heat level of the file data blocks, the heat level of the file data blocks corresponding to each lost data block is associated to obtain the comprehensive weight value of the data blocks to be repaired. Then, all the comprehensive weight values ​​of the data blocks to be repaired are sorted in descending order to establish a hierarchical repair task queue. Read the first task in the hierarchical repair task queue, retrieve the I / O path between the source storage node and the target storage node required by the task, collect the network bandwidth utilization and disk read / write queue length of the I / O path, generate the target I / O path load, compare the target I / O path load with the preset load threshold, and set the upper limit of the data transmission rate for repair after passing the test, thus forming a data reconstruction instruction set. Upon receiving the data reconstruction instruction set, and based on the set upper limit of data transmission rate, perform data reconstruction on the target storage node, generate a reconstructed data block verification sequence, and submit the location information and status change request of the new data block to the metadata node after the reconstructed data block verification sequence is consistent, and obtain the updated file system metadata table.

Images