A computer cloud storage management method for big data

Abstract

The application discloses a kind of computer cloud storage management methods for big data, belong to computer cloud storage technical field.The method includes: obtaining the multi-source big data set containing structured, unstructured and semi-structured data;Perform multi-redundancy data slice encoding processing, generate multi-redundancy data slice set;Multi-redundancy data slice set is executed data block strategy, and block granularity is dynamically adjusted according to access frequency and storage node load state, to generate adaptive block data set;Global metadata index construction is executed to adaptive block data set, to generate distributed hash index table;Based on distributed hash index table, perform load-aware scheduling processing, to generate storage node allocation scheme.The method improves the access efficiency of hot data through dynamic block granularity adjustment and load-aware scheduling, and optimizes the overall load balancing effect of storage cluster.

Description

Technical Field

[0001] This invention relates to the field of computer cloud storage technology, specifically to a computer cloud storage management method for big data. Background Technology

[0002] Cloud storage management methods in multi-source big data scenarios need to simultaneously process structured, unstructured, and semi-structured data, which are massive in scale and have complex access patterns. Existing storage management methods typically use a fixed-size block strategy to organize data and apply uniform indexing and scheduling rules to all data blocks. This static block approach does not distinguish between the access frequency of data blocks. High-frequency access hot data and low-frequency access cold data are treated with the same granularity, resulting in a large number of fragmented request conflicts and increased response latency when hot data is read concurrently, while cold data wastes index storage resources due to the retention of too many metadata entries. At the same time, existing block strategies do not consider the real-time load status of distributed storage nodes when determining the block size, and cannot perceive resource fluctuations such as CPU utilization and memory usage of nodes. This causes the block results to be disconnected from the actual processing capacity of the nodes, further exacerbating the uneven load among nodes. In terms of metadata indexing, conventional methods generally rely on centralized directories or simple consistent hash rings, which can only provide static location mapping and are difficult to integrate with the dynamic load information of nodes during data scheduling. Correspondingly, data block scheduling is mostly based on remaining storage capacity or fixed mapping relationships, lacking the ability to perceive real-time load indicators such as disk I / O queue depth and network bandwidth utilization. This leads to delayed scheduling decisions, causing some nodes to be overloaded while others remain idle, thus limiting the overall throughput and resource utilization efficiency of the storage cluster. The diverse redundancy coding requirements introduced by multi-source heterogeneous data further increase the difficulty of coordinating and optimizing block partitioning, indexing, and scheduling. Therefore, in cloud storage environments, it is necessary to address the problem of adaptively changing data block granularity with access frequency and node load under multi-redundant storage conditions, as well as the problem of precise scheduling jointly driven by globally distributed indexes and real-time load information. Summary of the Invention

[0003] The purpose of this invention is to provide a computer cloud storage management method for big data. For multi-source heterogeneous big data cloud storage scenarios, it solves the defects of existing technologies, such as fixed block granularity which cannot match data access characteristics and node load changes, and the separation of indexing system and load scheduling. It realizes adaptive block processing based on redundant coding and globally distributed load-aware scheduling, thereby improving the access efficiency and resource balance utilization of the storage system.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a computer cloud storage management method for big data, the method comprising: acquiring a multi-source big data set in a distributed storage cluster, the multi-source big data set including structured data, unstructured data, and semi-structured data; performing multi-redundant data slicing encoding processing on the multi-source big data set to generate a multi-redundant data slice set, wherein the multi-redundant data slice set is dynamically generated based on a preset redundancy threshold and data block size; performing a data partitioning strategy on the multi-redundant data slice set, dynamically adjusting the partitioning granularity according to the access frequency of each data block and the load status of storage nodes to generate an adaptive partitioned dataset; performing global metadata index construction processing on the adaptive partitioned dataset to generate a distributed hash index table, the distributed hash index table including the physical storage location identifier and logical path mapping relationship of each data block; and performing load-aware scheduling processing on the adaptive partitioned dataset based on the distributed hash index table to generate a storage node allocation scheme.

[0005] This invention combines multi-redundant slice encoding with adaptive block granularity adjustment to reduce redundancy overhead while ensuring data reliability. It also matches data block size with access frequency and node load, reducing input / output latency in high-frequency access scenarios. By constructing a distributed hash index table, a lightweight global view of metadata is achieved, supporting efficient data location and consistency maintenance. Combined with load-aware scheduling, data blocks are allocated to storage nodes with lower loads, effectively avoiding overload on hot nodes and balancing cluster resources.

[0006] As a preferred embodiment of the present invention, the step of implementing a data segmentation strategy on the multi-redundant data slice set, dynamically adjusting the segmentation granularity according to the access frequency of each data slice and the load status of the storage nodes, and generating an adaptive segmented dataset, specifically includes: performing access frequency statistical processing on each data slice in the multi-redundant data slice set to generate access frequency statistical values; comparing the access frequency statistical values ​​with a preset high-frequency threshold to identify high-frequency access data slices and low-frequency access data slices; performing fine-grained segmentation processing on the high-frequency access data slices to divide the slice size to a first preset granularity value; performing coarse-grained segmentation processing on the low-frequency access data slices to merge the slice size to a second preset granularity value, where the second preset granularity value is greater than the first preset granularity value; obtaining the CPU utilization and memory occupancy rate of each storage node to generate a node load status vector; performing a weighted calculation by fusing the node load status vector with the first preset granularity value and the second preset granularity value to generate dynamic segmentation granularity parameters; and re-segmenting the high-frequency access data slices and low-frequency access data slices based on the dynamic segmentation granularity parameters to generate the adaptive segmented dataset.

[0007] Through the above method, high-frequency data slices are further subdivided into smaller blocks, facilitating parallel access and rapid response; low-frequency data slices are merged into larger blocks, reducing metadata management pressure and disk seek overhead. The weighted mechanism of the fused load state vector allows the block granularity to adaptively adjust with changes in node load. When node load increases, the block granularity automatically increases to reduce scheduling fragmentation and ensure the overall throughput stability of the cluster. Preferably, in the fused weighted calculation, the dynamic block granularity of the high-frequency access data slices is negatively correlated with the node load state vector, thereby appropriately suppressing overly fine-grained block behavior under high load conditions and avoiding increased resource contention due to over-exploitation.

[0008] As a further preferred embodiment of the present invention, the step of performing global metadata index construction processing on the adaptive block dataset to generate a distributed hash index table specifically includes: extracting the block identifier and data digest value of each data block in the adaptive block dataset, wherein the data digest value is generated through hash operation; concatenating the block identifier and the data digest value to generate metadata key-value pairs for each data block; performing distributed consistent hash mapping processing on the metadata key-value pairs to allocate each metadata key-value pair to a corresponding index server node; constructing a local hash index sub-table on the index server node, wherein the local hash index sub-table uses the metadata key-value pairs as index entries; and performing global synchronization processing on the local hash index sub-tables of each index server node to generate the distributed hash index table.

[0009] This index construction method uses data digest values ​​to ensure the unique identification and integrity verification capability of block content, and adopts consistent hash mapping to make metadata evenly distributed and easy to expand. The globally synchronized distributed hash index table can quickly provide the association query from block to physical location, improving data location efficiency.

[0010] Furthermore, the step of globally synchronizing the local hash index sub-tables of each index server node to generate the distributed hash index table includes: obtaining the version number and timestamp of the local hash index sub-table of each index server node; performing difference comparison processing on each local hash index sub-table based on the version number and timestamp to identify newly added, modified, and deleted index entries; encapsulating the newly added, modified, and deleted index entries into incremental synchronization messages and broadcasting them to all index server nodes; after receiving the incremental synchronization message, each index server node performs an update operation on its own local hash index sub-table; and performing consistency verification processing on all updated local hash index sub-tables to generate the distributed hash index table.

[0011] The incremental synchronization mechanism based on version number and timestamp only needs to transmit the changed index entries, which greatly reduces the network overhead caused by metadata synchronization. At the same time, the eventual consistency check ensures the accuracy and consistency of the index views of each node, providing a reliable basis for subsequent scheduling.

[0012] As another preferred technical solution of the present invention, the step of performing load-aware scheduling processing on the adaptive block dataset based on the distributed hash index table to generate a storage node allocation scheme includes: querying the physical storage location identifier of each data block from the distributed hash index table; obtaining real-time load information of the storage nodes corresponding to each physical storage location, wherein the real-time load information includes disk I / O queue depth and network bandwidth utilization; calculating the load score of each storage node based on the disk I / O queue depth and network bandwidth utilization; sorting the data blocks to be scheduled in the adaptive block dataset by size to generate a queue of data blocks to be scheduled; and sequentially retrieving data blocks from the queue of data blocks to be scheduled in ascending order of load score and allocating them to the corresponding storage nodes.

[0013] By using both disk I / O queue depth and network bandwidth utilization as metrics to characterize node load, a comprehensive reflection of the storage node's I / O pressure and network transmission bottlenecks can be achieved. Data blocks are allocated from low to high load scores, prioritizing data writing or migration to nodes with lighter loads, thus achieving cluster-level load balancing and shortening data access response time.

[0014] Specifically, calculating the load score for each storage node based on the disk I / O queue depth and network bandwidth utilization includes: normalizing the disk I / O queue depth to a first interval to generate a disk load factor; normalizing the network bandwidth utilization to a second interval to generate a network load factor; assigning weight coefficients to the disk load factor and the network load factor, wherein the weight coefficients are dynamically updated based on historical scheduling records; and summing the product of the disk load factor and its corresponding weight coefficient, and the product of the network load factor and its corresponding weight coefficient, to generate the load score.

[0015] By eliminating differences in indicator dimensions through normalization and dynamically updating weight coefficients using historical scheduling records, the load score more accurately reflects the actual bottleneck level of disk and network resources in the current business scenario. Preferably, the weight coefficients are dynamically updated based on the ratio of disk input / output latency to network transmission latency in historical scheduling records, enabling the scheduling strategy to automatically favor disk-sensitive loads over network-sensitive loads, further improving overall data access performance.

[0016] As a preferred embodiment of the present invention, the step of performing multi-redundant data slicing encoding processing on the multi-source big data set to generate a multi-redundant data slice set includes: dividing each data file in the multi-source big data set into original data blocks of fixed size; performing erasure coding processing on each original data block to generate a first number of redundant data blocks; performing copy processing on each original data block to generate a second number of redundant data blocks, wherein the sum of the second number of redundant data blocks and the first number of redundant data blocks is equal to a preset redundancy threshold; and performing mixed sorting processing on the original data blocks, the first redundant data blocks, and the second redundant data blocks to generate the multi-redundant data slice set.

[0017] This hybrid redundancy strategy combines the advantages of low storage overhead of erasure coding and fast copy reading speed. While meeting the preset redundancy requirements, it can flexibly adjust the ratio of erasure coding to copies according to data importance or access characteristics, thus balancing storage efficiency and read performance.

[0018] As a preferred extension of the present invention, after generating the storage node allocation scheme, the method further includes: performing a health status detection process on each storage node in the storage node allocation scheme to generate a node health score; when the node health score is lower than a preset health threshold, migrating the data blocks on the storage node to other storage nodes with higher health scores; updating the physical storage location identifier of the migrated data blocks in the distributed hash index table; and rebroadcasting the updated distributed hash index table to all index server nodes.

[0019] By introducing a node health detection and automatic migration mechanism, we can respond promptly to faulty or degraded nodes, ensuring data availability and durability. After migration, the index table is updated and broadcast immediately to ensure that all index nodes can obtain the latest location mapping, avoiding interruption of read and write requests due to location invalidation.

[0020] The technical effects and advantages provided by the present invention in the above technical solution are as follows: When implementing a data partitioning strategy on a set of redundant data slices, the data slices are divided into high-frequency access data slices and low-frequency access data slices by statistically analyzing the access frequency of each data slice. High-frequency slices are partitioned into smaller first preset granularity values, while low-frequency slices are partitioned into larger second preset granularity values. At the same time, the CPU utilization and memory usage of each storage node are obtained to generate a node load state vector. This vector is then fused and weighted with the granularity value to obtain dynamic partitioning granularity parameters. The dynamic partitioning granularity of high-frequency slices is negatively correlated with the node load state vector. Based on this, high-frequency and low-frequency slices are re-partitioned to form an adaptive partitioned dataset. This series of processes incorporates data access characteristics and real-time node load into the block partitioning decision, enabling high-frequency data to be distributed with finer granularity. This supports multiple concurrent read operations distributed across different blocks, reducing single-block read conflicts and waiting times, and significantly improving the throughput of hot data. Low-frequency data is merged with coarse granularity, greatly compressing the number of metadata entries and reducing the storage overhead and query traversal cost of metadata indexes. After introducing negatively correlated node load correction, the block granularity is dynamically increased when the node load increases, reducing the frequency of block scheduling for that node, alleviating node pressure, and enabling data distribution to adaptively adjust with node status, thus enhancing the system's processing resilience under load fluctuations. When performing load-aware scheduling based on a distributed hash index table, the physical storage location identifier of each data block is queried from the index table to obtain the real-time load information of the corresponding storage node. This information includes the disk I / O queue depth and network bandwidth utilization. The disk load factor and network load factor are generated by normalizing the disk I / O queue depth and network bandwidth utilization. The weight coefficients of the two factors are dynamically updated based on the ratio of disk I / O wait time to network transmission latency in historical scheduling records. The weighted sum is then used to obtain the load score of each storage node. The adaptive block dataset to be scheduled is sorted by data block size, and data blocks are extracted and allocated to the corresponding storage nodes in order of load score from low to high. This scheduling mechanism tightly integrates the physical mapping capabilities provided by the global index with the multi-dimensional load status of nodes during runtime. Data blocks are preferentially directed to lightly loaded nodes, avoiding the allocation of new data blocks to nodes with accumulated disk queues or saturated network bandwidth. This alleviates the hotspot node phenomenon in the storage cluster and improves the overall load balancing level. The dynamic weighting coefficient enables the load score to adaptively reflect the bottleneck type of the current workload. When disk bottlenecks are prominent, the weight of the disk load factor is increased, and during network-sensitive phases, the weight of the network load factor is increased. The scheduling decision is more in line with the actual resource competition situation, effectively reducing the queuing latency and transmission time of data access operations, and improving the service quality and resource utilization efficiency of the cloud storage system. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart of a computer cloud storage management method for big data. Figure 2 This is a flowchart of the distributed hash index table construction process; Figure 3 This is a schematic diagram showing the distribution of distributed consistent hash mapping index server nodes; Figure 4 It is a probability density distribution map of the size of multiple redundant data slices; Figure 5 It is a curve showing the change of storage node health score over the detection period. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See Figure 1 This invention provides a computer cloud storage management method for big data. The method includes: acquiring a multi-source big data set from a distributed storage cluster, wherein the multi-source big data set includes structured data, unstructured data, and semi-structured data; performing multi-redundant data slicing encoding processing on the multi-source big data set to generate a multi-redundant data slice set, wherein the multi-redundant data slice set is dynamically generated based on a preset redundancy threshold and data block size; implementing a data partitioning strategy on the multi-redundant data slice set, dynamically adjusting the partitioning granularity according to the access frequency of each data block and the load status of storage nodes to generate an adaptive partitioned dataset; performing global metadata index construction processing on the adaptive partitioned dataset to generate a distributed hash index table, wherein the distributed hash index table contains the physical storage location identifier and logical path mapping relationship of each data block; and performing load-aware scheduling processing on the adaptive partitioned dataset based on the distributed hash index table to generate a storage node allocation scheme.

[0025] Example 1: In specific implementation, the data segmentation strategy is executed on the multi-redundant data slice set, and the segmentation granularity is dynamically adjusted according to the access frequency of each data block and the load status of the storage node. The process of generating an adaptive segmented dataset is achieved in the following way.

[0026] For each data slice in the multi-redundant data slice set, access frequency statistics are processed to generate corresponding access frequency statistics. In specific implementations, for each data slice, an access counter within a sliding time window is maintained. Whenever a computing node in the distributed storage cluster initiates a read request for a data slice, the count value of the access counter corresponding to that data slice within the time window increases by one unit. The length of the time window can be set according to the business scenario, for example, 600 seconds. After each time window period, the current value of the access counter is used as the access frequency statistics, and the access counter is reset to start the next statistical period. After generating the access frequency statistics, the access frequency statistics are compared with a preset high-frequency threshold to identify high-frequency and low-frequency access data slices. The preset high-frequency threshold is determined based on the distribution of historical access frequency statistics. In specific implementations, access frequency statistics of all data slices in the system over the past multiple time windows can be collected to form an access frequency statistics set. The upper quartile or 80th percentile of the access frequency statistics set can be calculated as the preset high-frequency threshold. When the access frequency statistics of a data slice are greater than the preset high-frequency threshold, the data slice is identified as a high-frequency access data slice; when the access frequency statistics of a data slice are less than or equal to the preset high-frequency threshold, the data slice is identified as a low-frequency access data slice.

[0027] Fine-grained segmentation is performed on the high-frequency access data slices, dividing the slice size to a first preset granularity value. This first preset granularity value is set based on the optimal transfer size for a single disk I / O operation in the distributed storage cluster; in a specific implementation, it can be set to 64 kilobytes. During fine-grained segmentation, the high-frequency access data slices are divided into fixed-length segments according to the first preset granularity value. If the remaining portion is less than the first preset granularity value, it is treated as an independent data block. Coarse-grained segmentation is performed on the low-frequency access data slices, merging the slice size to a second preset granularity value, which is larger than the first preset granularity value. This second preset granularity value is set based on the upper limit of the data block size for batch sequential reading in the distributed storage cluster; in a specific implementation, it can be set to 4 megabytes. When performing coarse-grained block processing on low-frequency access data slices, multiple consecutive low-frequency access data slices on the logical path are merged into a data block. The size of the merged data block does not exceed the second preset granularity value. If the merged data block exceeds the second preset granularity value, the excess part is re-divided into data blocks that conform to the second preset granularity value.

[0028] The CPU utilization and memory usage of each storage node are obtained to generate a node load state vector. In practice, a load collection agent is deployed on each storage node in the distributed storage cluster. This agent periodically reads the CPU utilization and memory usage of its assigned storage node, with a reading period of 5 seconds. CPU utilization is obtained by reading CPU time slice statistics provided by the operating system kernel, representing the proportion of time the CPU is in a non-idle state, with a value ranging from 0 to 1. Memory usage is obtained by reading the total memory and used memory provided by the operating system kernel, representing the ratio of used memory to total memory, with a value ranging from 0 to 1. For each storage node, the CPU utilization value obtained in the current collection period is recorded as... Memory usage is recorded as Generate node load state vector .

[0029] The node load state vector is fused and weighted with the first preset granularity value and the second preset granularity value to generate dynamic block granularity parameters. In specific implementation, the fusion and weighting calculation process includes the following: First, the CPU utilization rate in the node load state vector is... and memory usage The load values ​​are combined into a single scalar value through a weighted summation. The calculation formula is:

[0030] in, This is a weighting coefficient for central processing unit utilization, with a value ranging from 0 to 1. The settings are based on the historical task type distribution of the distributed storage cluster. When the proportion of computationally intensive tasks in the historical tasks exceeds 50%, ... The value should be greater than 0.5; when input-output intensive tasks account for more than 50% of the historical tasks, The value is less than 0.5. In one implementation, Set to 0.6. This is obtained through the above calculations. The value ranges from 0 to 1. and When both are 0, =0; when and When both are 1, The value is 1.

[0031] Then according to The first and second preset granularity values ​​are adjusted to generate dynamic block granularity parameters. For frequently accessed data slices, the dynamic block granularity parameters... The generation method is as follows:

[0032] in, Based on the basic block granularity value, for frequently accessed data slices The value is taken as the first preset granularity value; The negative correlation adjustment coefficient The value of makes the dynamic granularity of high-frequency access data slices negatively correlated with the node load state vector. In specific implementations... Set to -0.5. When When it increases, Compared to This reduces the size of the data, resulting in smaller data blocks. For infrequently accessed data slices, the dynamic block granularity parameter is generated using the same formula, but... The value is taken as the second preset granularity value. Setting it to 0.3 allows the dynamic granularity of low-frequency access data slices to increase moderately as node load increases, thereby reducing scheduling overhead.

[0033] In another specific implementation, the generation of dynamic block granularity parameters can also be applied to... Set upper and lower limits to limit extreme values ​​of granularity variation. For frequently accessed data slices, The lower limit is set to 0.5 times the first preset granularity value, and the upper limit is set to 1.2 times the first preset granularity value; for low-frequency access data slices, The lower limit is set to 0.8 times the second preset granularity value, and the upper limit is set to 1.5 times the second preset granularity value. If the calculated... If the value exceeds the specified range, the nearest boundary value will be used.

[0034] Based on the dynamic block granularity parameters, the high-frequency access data slices and low-frequency access data slices are re-segmented to generate the adaptive block dataset. In a specific implementation, the re-segmentation process is as follows: for each high-frequency access data slice, using... The value is used as the new segment size, and the high-frequency access data slice is re-segmented to generate multiple data blocks; for each low-frequency access data slice or existing low-frequency coarse-grained data block, the value is used as the new segment size. The value is used as the target size for merging or splitting, and the data block boundaries are adjusted so that the size of each data block is consistent with the target size. The values ​​are close to or equal to each other. All data blocks obtained after re-segmentation are aggregated to form the adaptive block dataset.

[0035] Example 2: In specific implementation, please refer to Figure 2 The process of performing global metadata index construction on the adaptive block dataset and generating a distributed hash index table is implemented in the following way.

[0036] Extract the block identifier and data digest value of each data block in the adaptive chunked dataset. In specific implementations, each data block is assigned a unique block identifier upon generation. The block identifier is formed by concatenating the file identifier of the original data file to which the data block belongs, the offset of the data block within the file, and the length of the data block, separated by an underscore character. Simultaneously, a hash operation is performed on the complete binary content of each data block to generate a data digest value. The hash operation uses the SHA-256 algorithm, and the data digest value is a 256-bit binary sequence, which is then converted into a 64-hexadecimal string representation. The data digest value uniquely represents the current content state of the data block; any modification to a data block will change the data digest value.

[0037] The block identifier and the data digest value are concatenated to generate metadata key-value pairs for each data block. In specific implementations, the key part of the metadata key-value pair consists of the block identifier, and the value part consists of the data digest value and additional attribute information. The additional attribute information includes the data block size, storage timestamp, and redundancy type identifier. The block identifier serves as the key, used for quickly locating the data block in the index structure; the data digest value, as the core content of the value, is used for data integrity verification; and the additional attribute information provides auxiliary parameters required for data block management. During the concatenation process, the key and value are joined using a predefined delimiter, with the colon character used as the delimiter, to form a complete metadata key-value pair string.

[0038] The metadata key-value pairs are processed using distributed consistent hash mapping, assigning each pair to a corresponding index server node. In this implementation, the distributed consistent hash mapping is based on a consistent hash ring. First, a ring-shaped hash space is constructed, with a range of 0 to... For each metadata key-value pair, the network address of each index server node is mapped to a location on the hash ring using the MD5 hash function, thus distributing the index server nodes across the hash ring. Then, for each metadata key-value pair, the key portion is extracted, and its hash value is calculated using the same MD5 hash function. This hash value is mapped to a location on the hash ring, and the nearest index server node to the hash value is located clockwise along the hash ring. This nearest index server node is then selected as the target index server node for storing the current metadata key-value pair. Through this mechanism, each metadata key-value pair is assigned to a unique index server node.

[0039] A local hash index sub-table is constructed on the index server node, using the metadata key-value pairs as index entries. In specific implementations, each index server node maintains a hash table structure in memory. The key of the hash table is the key portion of the metadata key-value pair, and the value is the value portion of the metadata key-value pair along with the corresponding timestamp information. After a metadata key-value pair is allocated to an index server node through distributed consistent hash mapping, the index server node inserts the key and value from the metadata key-value pair into the local hash index sub-table. If the same key already exists in the local hash index sub-table, the data digest value of the value portion is compared. If the data digest values ​​are different, it indicates that the data block content has been updated; in this case, the new value replaces the old value, and the corresponding timestamp information is updated. If the data digest values ​​are the same, only the timestamp information is updated. The local hash index sub-table provides each index server node with localized, fast metadata query capabilities.

[0040] The local hash index sub-tables of each index server node are globally synchronized to generate the distributed hash index table. The purpose of the global synchronization process is to ensure that all index server nodes ultimately hold the exact same index view. In specific implementation, the global synchronization process includes the following steps.

[0041] Retrieve the version number and timestamp of the local hash index sub-table for each index server node. In practice, each index server node maintains a monotonically increasing version number and a last update timestamp for its local hash index sub-table. The version number is initially set to 1, and it is incremented by 1 each time an insertion, modification, or deletion operation occurs in the local hash index sub-table, while the last update timestamp is updated to the current system time. The retrieval operation is carried out by the monitoring and coordination node in the system periodically sending status query requests to each index server node. Each index server node responds to the request, returning its own version number and last update timestamp of its local hash index sub-table. The monitoring and coordination node can be deployed on any of the index server nodes and is elected through a distributed election mechanism.

[0042] Based on the version number and timestamp, a difference comparison process is performed on each local hash index sub-table to identify newly added, modified, and deleted index entries. In specific implementation, after the monitoring and coordination node collects the version numbers and timestamps of all index server nodes, it selects the local hash index sub-table with the highest version number as the baseline version. If multiple index server nodes have the same version number and are all the highest, the timestamps of these index server nodes are further compared, and the local hash index sub-table with the latest timestamp is selected as the baseline version. After selecting the baseline version, the monitoring and coordination node compares each index entry in the local hash index sub-table of the baseline version with each entry in the local hash index sub-tables of other index server nodes. For index entries that exist in the baseline version but not in other index server nodes, they are identified as newly added index entries; for index entries that exist in both the baseline version and other index server nodes but have different values, they are identified as modified index entries; for index entries that do not exist in the baseline version but exist in other index server nodes, they are identified as deleted index entries. During the difference comparison process, the key part of the metadata key-value pair is used as the matching basis.

[0043] The newly added, modified, and deleted index entries are encapsulated into incremental synchronization messages and broadcast to all index server nodes. In specific implementations, incremental synchronization messages are encapsulated using a structured data format, containing a message type field and a message payload field. The message type field takes the value of one of the following: add, modify, or delete operation. For add operations, the message payload field contains the complete key and value of the newly added index entry; for modify operations, the message payload field contains the key of the modified index entry and its new value; for delete operations, the message payload field contains the key of the deleted index entry. The monitoring and coordination node broadcasts the encapsulated incremental synchronization messages to all index server nodes via a reliable multicast protocol. The reliable multicast protocol ensures that the incremental synchronization messages are correctly received by all online index server nodes. For temporarily offline index server nodes, the reliable multicast protocol caches the incremental synchronization messages, waiting for the index server nodes to come back online before delivery.

[0044] After receiving the incremental synchronization message, each index server node performs an update operation on its own local hash index sub-table. In specific implementation, the index server node parses the message type and message payload from the received incremental synchronization message and processes them accordingly: If it is an insert operation, the key and value in the message payload are inserted into the local hash index sub-table, and the local version number is incremented by 1; if it is a modify operation, the key in the message payload is searched for in the local hash index sub-table. If found, the original value is replaced with the new value in the message payload, and the local version number is incremented by 1; otherwise, it is processed as an insert operation; if it is a delete operation, the key in the message payload is searched for in the local hash index sub-table. If found, the index entry is deleted, and the local version number is incremented by 1; otherwise, it is ignored. After the update is complete, the index server node updates the last update timestamp of its local hash index sub-table to the current system time.

[0045] Consistency verification is performed on all updated local hash index sub-tables to generate the distributed hash index table. In practice, consistency verification is implemented using Merkle tree root value comparison. Each index server node constructs a Merkle tree based on the updated local hash index sub-tables. All leaf nodes of the Merkle tree are arranged lexicographically according to the keys of the index entries. Each leaf node stores a content digest value of an index entry, which is calculated using the SHA-256 hash function on the concatenated string of the index entry's key and value. The index server node calculates the hash values ​​of its internal nodes layer by layer from bottom to top, ultimately obtaining the Merkle tree root value. Each index server node broadcasts its own Merkle tree root value to the monitoring and coordination node, which compares all received Merkle tree root values. If all Merkle tree root values ​​are the same, it indicates that the local hash index sub-tables of all index server nodes have achieved global consistency. At this point, global synchronization is declared complete, and the set of all consistent local hash index sub-tables constitutes the distributed hash index table. If there are differences in the Merkle root values, the monitoring and coordination node will re-execute the difference comparison process and initiate incremental synchronization again until all Merkle root values ​​are consistent.

[0046] In another specific implementation, consistency verification can also be achieved by calculating the joint hash value of all local hash index sub-tables. All index server nodes serialize their respective local hash index sub-tables into ordered byte strings, calculate the hash value of the byte strings using the SHA-256 algorithm, and then broadcast their respective hash values ​​within the cluster. Each index server node collects all hash values; if all hash values ​​are equal, consensus is considered achieved. The formula for calculating the joint hash value is:

[0047] in, Represents the globally consistent joint hash value. This refers to the SHA-256 hash function. This represents a string concatenation operation. This represents the key of the first index entry after lexicographical sorting. This represents the value of the first index entry after sorting lexicographically. This represents the key of the second index entry after lexicographical sorting. This represents the value of the second index entry after sorting lexicographically. This represents the key of the nth index entry after lexicographical sorting. This represents the value of the nth index entry after lexicographical sorting, where n represents the total number of index entries in the local hash index subtable. This is calculated by comparing the values ​​of each node. If they are the same, complete the consistency check and generate a distributed hash index table.

[0048] See Figure 3 The scatter plot illustrates the distribution of the key portion of the metadata key-value pairs in the distributed hash index table—that is, the hash value of the block identifier—on the hash ring to the corresponding index server nodes. The horizontal axis represents the hash value of the key portion, expressed in 16-bit decimal, ranging from 0 to approximately 70,000. The vertical axis represents the index server node number, which includes four nodes: 0, 1, 2, and 3. Different colored scatter points represent the metadata key-value pairs assigned to different index server nodes.

[0049] The diagram shows that the hash value range is clearly divided into four consecutive and non-overlapping segments, corresponding to index server nodes 1, 2, 3, and 0, respectively. Specifically, all key-value pairs in the hash value range from 0 to approximately 16,000 are assigned to index server node 1; those from approximately 16,000 to 33,000 are assigned to index server node 2; those from approximately 33,000 to 48,000 are assigned to index server node 3; and those from approximately 48,000 to the maximum hash value are assigned to index server node 0. The hash value ranges assigned to each node are continuous and cover the entire hash ring range without any overlap.

[0050] This distribution conforms to the principle of distributed consistent hash mapping described in Example 2, namely, after calculating the key hash value using a hash function, the nearest index server node is located along the hash ring in a clockwise direction, completing the even distribution of metadata key-value pairs among the index server nodes. The even distribution of each point in the figure within its corresponding hash interval indicates that the node mapping of the consistent hash ring is reasonable, and the load distribution of the index server nodes is relatively balanced, which is beneficial for improving the query efficiency of the global metadata index and the scalability of the system.

[0051] Example 3: In specific implementation, the process of performing load-aware scheduling on the adaptive block dataset based on the distributed hash index table and generating a storage node allocation scheme is achieved in the following way.

[0052] The physical storage location identifier for each data block is retrieved from the distributed hash index table. In a specific implementation, the distributed hash index table is distributed across all index server nodes. When a query is performed, the scheduler sends a query request to the index server node cluster. For each data block in the adaptive block dataset, the scheduler extracts the block identifier and sends it as the query key to the index server node cluster. The index server node cluster locates the target index server node holding the index entry in the distributed hash index table based on the hash value of the block identifier. The target index server node retrieves its local hash index sub-table and returns the value portion corresponding to the block identifier. The scheduler parses the physical storage location identifier field from the value portion. The physical storage location identifier contains the storage node identifier and the storage path information within the storage node. In one implementation, the physical storage location identifier adopts the Uniform Resource Identifier (URI) format, where the format string is composed of the storage node's network address, port number, and absolute path. The scheduler extracts the storage node identifier by parsing the URI.

[0053] The system acquires real-time load information for storage nodes corresponding to each physical storage location. This real-time load information includes disk I / O queue depth and network bandwidth utilization. In specific implementations, each storage node in the distributed storage cluster runs a load information collection agent. This agent collects real-time load metrics for its storage node at a fixed interval of 2 seconds. Disk I / O queue depth is obtained by reading disk statistics provided by the operating system. In Linux, this is achieved by reading the 11th field of the pseudo-filesystem file located at " / sys / block / device_name / stat" to obtain the number of incomplete I / O requests. This number of incomplete I / O requests is used as the raw value of the disk I / O queue depth. Network bandwidth utilization is obtained by differentially calculating the number of bytes sent and received by the storage node's network interface. The calculation method is as follows: within a collection period, the sum of the increments in the number of bytes sent and received by the network interface is recorded. This sum of increments is divided by the collection period duration to obtain the current network throughput. Then, the current network throughput is divided by the maximum bandwidth value of the network interface to obtain the network bandwidth utilization. The maximum bandwidth value of the network interface is obtained from the network interface configuration. At the start of each collection cycle, the scheduler sends a load information query request to the load information collection agents of all storage nodes. Each load information collection agent encapsulates its collected disk I / O queue depth and network bandwidth utilization in a response message and returns it to the scheduler.

[0054] The load score for each storage node is calculated based on the disk I / O queue depth and network bandwidth utilization. In a specific implementation, the calculation of the load score includes the following steps.

[0055] The disk I / O queue depth is normalized to a first interval to generate a disk load factor. The first interval is a closed interval from 0 to 1. The normalization process is as follows: the original value of the disk I / O queue depth is divided by a preset maximum queue depth threshold to obtain the disk load factor. The preset maximum queue depth threshold is set according to the specifications of the disk devices equipped on the storage node. When the storage node uses an enterprise-grade solid-state drive, the maximum queue depth threshold is set to 128; when the storage node uses a hard disk drive, the maximum queue depth threshold is set to 32. The disk load factor is expressed as... , The calculation uses division. When the original disk input / output queue depth is greater than or equal to the maximum queue depth threshold, The value is 1.

[0056] The network bandwidth utilization rate is normalized to a second interval to generate the network load factor. The second interval is a closed interval from 0 to 1. Since the network bandwidth utilization rate itself is the ratio of the current network throughput to the maximum bandwidth, and its value ranges from 0 to 1, the network bandwidth utilization rate is directly used as the network load factor. The network load factor is expressed as... , The value is the collected network bandwidth utilization rate.

[0057] Weighting coefficients are assigned to the disk load factor and network load factor, and these coefficients are dynamically updated based on historical scheduling records. In practice, a historical scheduling record database is maintained. Each record in the database contains the disk I / O latency and network transmission latency measured after a data block allocation operation is completed. Disk I / O latency is defined as the time interval from when a data block write request is issued to when the storage node returns a write completion confirmation. Network transmission latency is defined as the one-way transmission delay incurred by the data block during network transmission. After each data block scheduling task is completed, the scheduling manager appends the disk I / O latency and network transmission latency of this scheduling task to the historical scheduling record database and updates the weighting coefficients.

[0058] The dynamic update method for the weighting coefficients is as follows: All scheduling records within the most recent time window are extracted from the historical scheduling record database, with the time window length set to 600 seconds. The average disk I / O wait time of all extracted scheduling records is calculated and denoted as... Calculate the average network transmission delay across all extracted scheduling records, denoted as . .according to and The ratio determines the weighting coefficient. The weighting coefficient of the disk load factor is denoted as... The weighting coefficient of the network load factor is denoted as . The value is obtained through and The functional relationship determines: when Greater than When this occurs, it indicates that disk I / O latency dominates the overall latency, and increasing this value will improve the situation. The value is used to increase the impact of the disk load factor on the load score; when Greater than When this occurs, it indicates that network transmission latency is dominant, and reducing [the frequency / reduction] is appropriate. The value is used to enhance the influence of the network load factor. In one specific implementation, The calculation does not rely on a fixed formula, but is achieved through a piecewise mapping function: if and The ratio is greater than 2.0. Set to 0.8; if the ratio is between 1.0 and 2.0, Set to 0.6; if the ratio is between 0.5 and 1.0, Set to 0.4; if the ratio is less than 0.5, Set it to 0.2. This way, the weighting coefficient is dynamically updated based on the ratio of disk I / O latency to network transmission latency in historical scheduling records.

[0059] The load score is generated by summing the products of the disk load factor and its corresponding weight coefficient, and the products of the network load factor and their corresponding weight coefficients. The formula for calculating the load score is as follows:

[0060] in, This indicates the load score of the storage node. This represents the weighting coefficient of the disk load factor. Indicates disk load factor. This represents the network load factor. The value ranges from 0 to 1. A higher value indicates a greater overall load on the storage node. The above summation operation is performed once for each storage node in each collection cycle to generate a load score for each storage node.

[0061] The data blocks to be scheduled in the adaptive block dataset are sorted by size to generate a queue of data blocks to be scheduled. In specific implementations, the scheduling manager selects data blocks that need to be reallocated or initially allocated from the adaptive block dataset, forming a set of data blocks to be scheduled. For each data block to be scheduled, the size attribute of the data block is read. The size attribute of the data block is recorded in the metadata of each data block when the adaptive block dataset is generated and can be retrieved through a distributed hash index table. All data blocks to be scheduled are arranged in descending order of data block size, that is, the largest data block is placed at the beginning and the smallest data block is placed at the end, forming the queue of data blocks to be scheduled.

[0062] Data blocks are sequentially retrieved from the queue of data blocks to be scheduled, according to their load scores from low to high, and allocated to the corresponding storage nodes. In practice, the scheduling manager maintains a list of storage node load scores, storing the node identifier and current load score of each storage node. Initially, the load scores in the list are the most recently calculated load scores. Starting from the head of the queue of data blocks to be scheduled, the scheduling manager retrieves one data block at a time. For each retrieved data block, the scheduling manager iterates through the storage node load score list, finding the storage node with the lowest current load score as the target storage node. After determining the target storage node, the scheduling manager allocates the data blocks to the target storage node, generating a storage node allocation record containing the mapping relationship between the data block identifier and the target storage node identifier. After completing one allocation, the scheduling manager updates the load score of the target storage node using an estimation method: adding the expected disk I / O queue depth increment based on the allocated data block size to the original value of the target storage node's disk I / O queue depth, recalculating the target storage node's disk load factor and load score, and updating the storage node load score list. Then, the next data block in the queue of data blocks to be scheduled is processed until all data blocks to be scheduled have been allocated. Finally, all storage node allocation records constitute the storage node allocation scheme.

[0063] Example 4: In specific implementation, the process of performing multi-redundant data slice encoding on the multi-source big data set to generate a multi-redundant data slice set is achieved in the following way.

[0064] Each data file in the multi-source big data set is divided into fixed-size raw data blocks. In specific implementations, the data files in the multi-source big data set are read one by one. The data file types include structured data files, unstructured data files, and semi-structured data files. For each data file, a fixed-length block segmentation method is used, dividing the byte sequence of the data file from the beginning of the file into multiple raw data blocks. The size of each raw data block is set to a fixed value. The fixed size of the raw data block is set according to the optimal transmission size for the input / output operations of the storage nodes in the distributed storage cluster. In one implementation, the raw data block size is set to 64 megabytes. When the end of the data file is reached, if the number of remaining bytes is less than the fixed size, the remaining bytes constitute a separate raw data block without padding. After each raw data block is generated, it is assigned a raw data block identifier, which is composed of the hash value of the source data file path, the sequence number of the raw data block in the file, and the block timestamp, connected by underscores.

[0065] Erasure coding is performed on each original data block to generate the first redundant data block. In specific implementations, the erasure coding process is implemented using the Reed-Solomon coding algorithm. The Reed-Solomon coding algorithm treats each original data block as a data object, divides the data object into k data segments, and then generates m check segments through coding calculations. The total number of segments after coding is k+m. Here, k represents the number of data segments, and m represents the number of check segments, which is the number of first redundant data blocks. The values ​​of k and m are determined based on a preset redundancy threshold and data protection strength requirements. After erasure coding of a single original data block, the number of check segments generated, m, is the number of first redundant data blocks. In one specific implementation, k is set to 6 and m is set to 2, indicating that each original data block is divided into 6 data segments and 2 check segments are generated. The entire original data block can be recovered if no more than 2 segments are damaged. During erasure coding, the encoding operation is performed independently for each original data block. Starting from the first byte of the original data block, encoding is performed in groups of k data segments, generating m parity segments in each group. If the size of the original data block is not divisible by k, the codec pads the last data segment with zero bytes to meet the calculation requirements. The padding is removed during subsequent decoding and recovery. Each group of m parity segments is marked as an erasure coding redundant data block, and the corresponding original data block identifier and group number are recorded.

[0066] For each original data block, a replication process is performed to generate a second redundant data block. The sum of the second redundant data block number and the first redundant data block number equals a preset redundancy threshold. In specific implementations, the replication process is achieved by copying the complete original data block byte-by-byte; the number of copies is the second redundant data block number. The value of the second redundant data block number is determined by both the preset redundancy threshold and the first redundant data block number. The preset redundancy threshold is denoted by the symbol... The number of the first redundant data blocks is indicated by the symbol. The number of second redundant data blocks is indicated by the symbol. This indicates that the three satisfy the following quantitative relationship:

[0067] in, This indicates the preset redundancy threshold. It is a positive integer greater than or equal to 2. The specific value is set according to the data file's classification level: when the data file is marked as high availability, Set to 4; when the data file is marked as standard level, Set to 3; when the data file is marked as normal level, Set to 2. This indicates the number of the first redundant data blocks generated through erasure coding. The value is the number of check fragments m in the erasure coding algorithm parameters. This indicates the number of second redundant data blocks generated through the replication process. The value is obtained through Calculated, and It is a positive integer greater than or equal to 1. In determining... It is necessary to ensure that Less than ,make The condition of being greater than or equal to 1 must be met. In one specific implementation, for standard-level data files, The value is 3. If the value is 2, then A value of 1 indicates that two erasure code check blocks and one complete replica block are generated for each original data block. The replication process copies each original data block byte by byte. Each copy serves as a second redundant data block, and is marked with the corresponding original data block identifier and copy number.

[0068] The original data blocks, the first redundant data blocks, and the second redundant data blocks are mixed and sorted to generate the multi-redundant data slice set. In specific implementations, the mixed sorting process places all the original data blocks generated in the above steps, the first redundant data blocks generated through erasure coding, and the second redundant data blocks generated through replication into the same set. Then, each data block is globally sorted according to the lexicographical order of its block identifier. The block identifier includes the original data block identifier and a redundancy type suffix during generation. The block identifier suffix for the original data block is "_orig", the block identifier suffix for the first redundant data block is "_ec_group_serial_checksum_number", and the block identifier suffix for the second redundant data block is "_rp_replica_serial_number". During sorting, the original data block identifiers are first sorted in ascending order by their primary identifier; if the primary identifiers are the same, they are sorted in ascending order by the lexicographical order of their suffixes. The resulting linear sequence after sorting is the multi-redundant data slice set. Each element in the multi-redundant data slice set is a data slice, which contains a block identifier, a slice type label, and the actual data content within the slice. Through the above hybrid sorting, the original data blocks and various redundant data blocks are logically organized in an orderly manner, which facilitates subsequent block and indexing operations.

[0069] See Figure 4 In the figure, the horizontal axis represents the data slice size in megabytes (MB), and the vertical axis represents the probability density, describing the distribution characteristics of data slices of different sizes. In the legend, the blue solid curve represents the original data block, the red dashed curve represents the first redundant data block (erasure code check block), and the green dotted line represents the second redundant data block (replica block). According to Example 4, this figure reflects the specific distribution of data block size in the multi-redundant data slice set generated after the multi-source big data set has undergone multi-redundant data slice encoding processing.

[0070] First, the original data block sizes, represented by the blue solid line, are concentrated in the range of approximately 62MB to 68MB, with a peak at approximately 65MB. This aligns with the fixed-size original data block size of around 64MB set in the embodiment. During final processing, the remaining portion forms a smaller number of boundary blocks, resulting in a slight expansion of the distribution. This distribution reflects the step in Embodiment 4 of "dividing the data file into fixed-size original data blocks, with the size set at 64 megabytes."

[0071] Secondly, the size of the first redundant data block (erasure code check block), represented by the red dashed line, is mainly concentrated between approximately 8MB and 12MB, with a peak of about 10MB, and its probability density is significantly higher than other areas. This reflects the process of dividing each 64MB original data block into k=6 data segments using the Reed-Solomon coding algorithm, and then generating m=2 check segments through encoding. This results in a single check segment being significantly smaller than the original block size, approximately 1 / 6 of the original data block size, which is consistent with the fragmentation characteristics of erasure code encoding in the embodiment.

[0072] Finally, the size distribution of the second redundant data block (replica block), represented by the green dotted line, highly overlaps with the original data block, with a peak value close to 65MB, and the probability density curves almost completely overlap. This indicates that the second redundant data block generated by the replication process is a complete byte-level copy of the original data block, with the same size as the original data block, and the number determined by the redundancy threshold.

[0073] Example 5: In practice, after generating the storage node allocation scheme, a health status detection process is performed on each storage node in the storage node allocation scheme, and the process of generating a node health score is implemented in the following way.

[0074] In practice, health status detection is initiated periodically by the monitoring and coordination node in the distributed storage cluster, with a detection cycle set to 30 seconds. At the end of each detection cycle, the monitoring and coordination node sends a health status query request to each storage node listed in the storage node allocation scheme. Each storage node runs a health status collection agent. Upon receiving the query request, the agent collects multiple health indicators for its storage node. These indicators include the storage node's CPU temperature, memory error correction code error count, number of bad disk blocks, disk self-monitoring analysis and reporting technology status value, and network interface link status value. The CPU temperature is obtained by reading the embedded controller register, in degrees Celsius. The memory error correction code error count is obtained by reading the memory error counter in the operating system kernel, representing the cumulative number of correctable errors since system startup. The number of bad disk blocks is obtained by sending a self-monitoring analysis and reporting technology query command to the disk device, extracting the bad block count attribute. The disk self-monitoring analysis and reporting technology status value is obtained by parsing the disk self-monitoring analysis and reporting technology status return code; a status value of 0 indicates normal disk health, while a status value of 1 indicates a warning condition. The network interface link status value is obtained by reading the operating system's network interface management information base. A link status value of 1 indicates that the network link is connected, and a link status value of 0 indicates that the network link is disconnected.

[0075] The health status collection agent encapsulates the above five health indicators into health indicator vectors and returns them to the monitoring and coordination node via a response message. After receiving the health indicator vectors from all storage nodes, the monitoring and coordination node quantifies and evaluates the health indicator vector of each storage node, generating a node health score. The calculation process for the node health score is as follows: a health baseline range and mapping function are set for each health indicator, mapping the current measurement value to a standard scoring interval. This standard scoring interval is a closed interval from 0 to 100, where 100 represents the best health status and 0 represents the worst health status.

[0076] The mapping function for CPU temperature is set as follows: when the CPU temperature is below 70 degrees Celsius, the score for this item is 100 points; when the CPU temperature is between 70 and 90 degrees Celsius, the score for this item decreases linearly, and the formula is score = 100 - (temperature value - 70) * 5; when the CPU temperature is above 90 degrees Celsius, the score for this item is 0 points.

[0077] The mapping function for the number of memory error correction codes is set as follows: when the number of memory error correction codes is 0, the score for this item is 100 points; when the number of memory error correction codes is between 1 and 10, the score for this item is 100 - number of errors * 10; when the number of memory error correction codes is greater than 10, the score for this item is 0 points.

[0078] The mapping function for the number of bad blocks on the disk is set as follows: when the number of bad blocks on the disk is 0, the score for this item is 100; when the number of bad blocks on the disk is between 1 and 50, the score for this item is 100 - number of bad blocks * 2; when the number of bad blocks on the disk is greater than 50, the score for this item is 0.

[0079] The mapping function for the disk self-monitoring analysis and reporting technology status value is set as follows: when the disk self-monitoring analysis and reporting technology status value is 0, the score for this item is 100 points; when the disk self-monitoring analysis and reporting technology status value is 1, the score for this item is 20 points.

[0080] The mapping function for network interface link status values ​​is set as follows: when the network interface link status value is 1, the score for this item is 100 points; when the network interface link status value is 0, the score for this item is 0 points.

[0081] The final calculation formula for the node health score is the weighted sum of the scores of the above five indicators. The weighting coefficients are pre-set based on the degree of impact of each indicator on the data availability of the storage node. In one implementation, the weighting coefficient vector is set as (0.2, 0.15, 0.3, 0.2, 0.15), corresponding to the CPU temperature, memory error correction code error count, number of bad disk blocks, disk self-monitoring analysis and reporting technical status value, and network interface link status value, respectively. The calculation formula for the node health score is:

[0082] in, This represents the node's health score, with a value ranging from 0 to 100. The weight of the CPU temperature score is set to 0.2. The central processing unit temperature fraction is obtained through the aforementioned mapping function; The weight for the score representing the number of memory error correction codes is set to 0.15; This represents the score for the number of memory error correction codes. The weight for the score indicating the number of bad blocks on the disk is set to 0.3; This represents the number of bad blocks on the disk (in fractions). The weight of the disk self-monitoring analysis and reporting technical status value score is set to 0.2; This indicates the score of the disk self-monitoring analysis and reporting technical status value; The weight representing the network interface link state value score is set to 0.15; This represents the network interface link status score. Each score ranges from 0 to 100.

[0083] When a node's health score falls below a preset health threshold, data blocks on that storage node are migrated to other storage nodes with higher health scores. In this implementation, the preset health threshold is set to 50 points, based on the fact that when a node's health score is below 50, at least one key health indicator has severely deteriorated, significantly increasing the risk of data read or write failures. The monitoring and coordination node iterates through the node health scores of all storage nodes, marking those with scores below 50 as nodes to be migrated. For each node to be migrated, the monitoring and coordination node queries a distributed hash index table to retrieve all data blocks whose physical storage location identifiers contain the identifier of the node to be migrated, forming a list of data blocks to be migrated. Simultaneously, the monitoring and coordination node selects storage nodes with health scores above 80 from the current node health score list as candidate target nodes. The selection threshold for candidate target nodes is set to 80 points to ensure that data is migrated to healthy storage nodes. For each data block to be migrated, the monitoring and coordination node selects the storage node with the highest health score from the candidate target nodes as the migration target node. If multiple nodes have the highest scores, the node with the largest remaining available storage space is selected. The scheduling manager copies the data blocks to be migrated from the node to be migrated to the target node, and deletes the original data blocks from the node to be migrated after confirming that the copy is complete.

[0084] The physical storage location identifier of the migrated data block in the distributed hash index table is updated. In specific implementations, when the data block migration is complete, the scheduler generates a new physical storage location identifier for each migrated data block. The new physical storage location identifier includes the target node identifier and the storage path on the target node. The scheduler sends an index update request to the index server node cluster, which includes the block identifier of the migrated data block and the new physical storage location identifier. The index server node cluster locates the corresponding index entry in the distributed hash index table based on the block identifier, updates the physical storage location identifier field in the index entry with the new physical storage location identifier, increments the entry version number by 1, and refreshes the timestamp.

[0085] The updated distributed hash index table is rebroadcast to all index server nodes. In practice, after updating the index entries, the index server node cluster triggers a global synchronization process. The monitoring and coordination node re-executes the difference comparison process, generating an incremental synchronization message containing the modified index entries. The modification operation message payload in the incremental synchronization message includes the key of the migrated data block and its modified value. The monitoring and coordination node broadcasts the incremental synchronization message to all index server nodes via a reliable multicast protocol. Each index server node receives the message, updates its local hash index sub-table, and performs consistency verification, ultimately bringing the distributed hash index tables of all index server nodes to a consistent state.

[0086] See Figure 5 In the graph, the horizontal axis represents the cycle number of the health status detection, with values ​​ranging from 1 to 500. The vertical axis represents the health score of the storage node, in points, ranging from 0 to 100. The graph contains five curves, corresponding to the health score trends of storage nodes A, B, C, D, and E, respectively, as well as a preset health threshold line set at 50 points.

[0087] The health score curve of storage node A is represented by a solid red line. Initially, the health score was around 90 points. As the detection cycle increased, the health score showed a clear and continuous downward trend. Around the 350th detection cycle, it fell below the preset health threshold of 50 points and continued to decline to around 30 points in subsequent detection cycles, indicating that the health status of node A was gradually deteriorating and posing a significant risk.

[0088] The health score curve of storage node B is represented by the blue dashed line. The overall fluctuation range is between 80 and 90 points. The score is relatively stable and always higher than the health threshold, indicating that node B maintains a good health status.

[0089] The health score curve of storage node C is represented by an orange dotted line. The health score fluctuates between 75 and 80 points, showing a relatively stable trend and is significantly higher than the threshold, indicating that node C is in a relatively normal operating state.

[0090] The health score curve for storage node D is represented by a green dotted line. The score fluctuates between approximately 65 and 70 points, showing slight fluctuations overall and consistently remaining above the threshold. This indicates that node D's health status is relatively stable but lower than that of nodes B and C.

[0091] The health score curve of storage node E is represented by a purple dotted line. The health score fluctuates around 90 points, is the highest and relatively stable, and is consistently far above the health threshold, indicating that node E is in the best health state.

[0092] The preset health threshold is represented by a gray dashed line at the 50-point mark on the vertical axis, serving as a critical boundary for determining whether a storage node requires data migration. According to the diagram, when a storage node's health score falls below this threshold, the data migration process must be initiated.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A computer cloud storage management method for big data, characterized in that, The method includes: Obtain a multi-source big data set from a distributed storage cluster, wherein the multi-source big data set includes structured data, unstructured data, and semi-structured data; The multi-source big data set is subjected to multi-redundant data slicing encoding processing to generate a multi-redundant data slice set, which is dynamically generated based on a preset redundancy threshold and data block size. A data segmentation strategy is executed on the multi-redundant data slice set, and the segmentation granularity is dynamically adjusted according to the access frequency of each data block and the load status of the storage node to generate an adaptive segmented dataset. A global metadata index construction process is performed on the adaptive block dataset to generate a distributed hash index table, which contains the physical storage location identifier and logical path mapping relationship of each data block; Based on the distributed hash index table, load-aware scheduling is performed on the adaptive block dataset to generate a storage node allocation scheme.

2. The computer cloud storage management method for big data according to claim 1, characterized in that, The step of implementing a data segmentation strategy on the multi-redundant data slice set, dynamically adjusting the segmentation granularity based on the access frequency of each data block and the storage node load status, and generating an adaptive segmented dataset includes: Perform access frequency statistics processing on each data slice in the multi-redundant data slice set to generate access frequency statistics; The access frequency statistics are compared with a preset high-frequency threshold to identify high-frequency access data slices and low-frequency access data slices. Fine-grained block processing is performed on the high-frequency access data slices to divide the slice size to a first preset granularity value; The low-frequency access data slices are subjected to coarse-grained block processing, and the slice sizes are merged into a second preset granularity value, which is greater than the first preset granularity value. Obtain the CPU utilization and memory usage of each storage node, and generate a node load status vector; The node load state vector is fused and weighted with the first preset granularity value and the second preset granularity value to generate dynamic block granularity parameters; The high-frequency access data slice and low-frequency access data slice are re-segmented based on the dynamic block granularity parameter to generate the adaptive block dataset.

3. The computer cloud storage management method for big data according to claim 2, characterized in that, In the fusion weighted calculation, the dynamic block granularity of the high-frequency access data slice is negatively correlated with the node load state vector.

4. The computer cloud storage management method for big data according to claim 1, characterized in that, The step of performing global metadata index construction processing on the adaptive block dataset to generate a distributed hash index table includes: Extract the block identifier and data digest value of each data block in the adaptive block dataset, wherein the data digest value is generated by hash operation; The block identifier and the data digest value are concatenated to generate metadata key-value pairs for each data block; Perform distributed consistent hash mapping on the metadata key-value pairs, and assign each metadata key-value pair to the corresponding index server node; A local hash index sub-table is constructed on the index server node, and the local hash index sub-table uses the metadata key-value pairs as index entries; The local hash index sub-tables of each index server node are globally synchronized to generate the distributed hash index table.

5. A computer cloud storage management method for big data according to claim 4, characterized in that, The step of globally synchronizing the local hash index sub-tables of each index server node to generate the distributed hash index table includes: Retrieve the version number and timestamp of the local hash index sub-table for each index server node; Based on the version number and timestamp, perform difference comparison processing on each local hash index sub-table to identify newly added, modified, and deleted index entries; The newly added, modified, and deleted index entries are encapsulated into incremental synchronization messages and broadcast to all index server nodes; After receiving the incremental synchronization message, each index server node performs an update operation on its own local hash index sub-table; Perform consistency verification on all updated local hash index sub-tables to generate the distributed hash index table.

6. The computer cloud storage management method for big data according to claim 1, characterized in that, The step of performing load-aware scheduling processing on the adaptive block dataset based on the distributed hash index table to generate a storage node allocation scheme includes: Retrieve the physical storage location identifier of each data block from the distributed hash index table; Obtain real-time load information of storage nodes corresponding to each physical storage location, wherein the real-time load information includes disk input / output queue depth and network bandwidth utilization; The load score for each storage node is calculated based on the disk I / O queue depth and network bandwidth utilization. The data blocks to be scheduled in the adaptive block dataset are sorted by size to generate a queue of data blocks to be scheduled. Data blocks are retrieved from the queue of data blocks to be scheduled in order of load score from low to high and allocated to the corresponding storage nodes.

7. A computer cloud storage management method for big data according to claim 6, characterized in that, The calculation of the load score for each storage node based on the disk I / O queue depth and network bandwidth utilization includes: The disk input / output queue depth is normalized to the first interval to generate a disk load factor; The network bandwidth utilization is normalized to the second interval to generate the network load factor; Weighting coefficients are assigned to the disk load factor and network load factor, and these weighting coefficients are dynamically updated based on historical scheduling records. The load score is generated by summing the product of the disk load factor and its corresponding weight coefficient, and the product of the network load factor and its corresponding weight coefficient.

8. A computer cloud storage management method for big data according to claim 7, characterized in that, The weighting coefficients are dynamically updated based on the ratio of disk input / output latency to network transmission latency in historical scheduling records.

9. A computer cloud storage management method for big data according to claim 1, characterized in that, The step of performing multi-redundant data slicing encoding processing on the multi-source big data set to generate a multi-redundant data slice set includes: Each data file in the multi-source big data set is divided into raw data blocks of a fixed size; Perform erasure coding on each original data block to generate the first number of redundant data blocks; For each original data block, a copy process is performed to generate a second redundant data block. The sum of the second redundant data block and the first redundant data block is equal to a preset redundancy threshold. The original data block, the first redundant data block, and the second redundant data block are mixed and sorted to generate the multi-redundant data slice set.

10. A computer cloud storage management method for big data according to claim 1, characterized in that, After generating the storage node allocation scheme, the method further includes: Perform health status detection processing on each storage node in the storage node allocation scheme to generate a node health score; When the health score of a node is lower than a preset health threshold, the data blocks on that storage node are migrated to other storage nodes with higher health scores. Update the physical storage location identifier of the migrated data block in the distributed hash index table; The updated distributed hash index table is rebroadcast to all index server nodes.

Images