Block chain collaborative storage method and system based on weighted consistency hash

Abstract

The invention discloses a block chain collaborative storage method and system based on weighted consistency Hash, and relates to the technical field of distributed data management, and the method comprises the steps: electing an anchor node in a group; the anchor node allocates the number of virtual nodes according to the available storage capacity of each physical node, virtual node coordinates are generated, and a consistent hash ring and a mapping table are constructed; setting a backup number parameter n, calculating n hash values for a new block according to a new block identifier and a backup sequence number, performing clockwise search and duplicate removal to supplement n target physical nodes, and distributing, writing and recording mapping of the block and a node set; and when the detection node fails, updating the hash ring, screening an affected block, recalculating a target node, determining an incremental node set, complementing and writing from the existing copy, and updating the mapping. According to the technical scheme of the invention, load balancing and copy discrete distribution driven by capacity weight are realized, the node change migration volume is reduced, the write availability and the repair efficiency are improved, and the ledger continuity is ensured.

Description

Technical Field

[0001] This invention relates to the field of distributed data management technology, and in particular to a blockchain collaborative storage method based on weighted consistent hashing. Background Technology

[0002] Blockchain systems typically require reliable storage and redundant backups of block data to support long-term availability and consistency verification of ledger data. In scenarios involving large-scale nodes and high-throughput writes, relying solely on full-node full storage leads to high storage costs and low scalability. Therefore, engineering often employs a distributed storage strategy based on consistent hashing: mapping block identifiers to a hash space and determining the storage node for each block on a hash ring, while simultaneously improving fault tolerance through a multi-replica mechanism.

[0003] However, existing block storage schemes based on consistent hashing still have significant shortcomings under the conditions of heterogeneous and dynamically changing blockchain nodes. First, the available storage capacity of blockchain physical nodes often varies greatly. If a fixed number of virtual nodes or a static allocation method is used, it is easy to cause a mismatch between the load distribution on the hash ring and the node capacity, resulting in some nodes having excessive storage pressure while others have idle resources, affecting the overall storage utilization and write stability. Second, multi-replica writes usually need to meet the constraint that replicas are distributed across different physical nodes. However, when hash ring positioning and node deduplication are not perfect, replicas may end up on the same physical node or there may be insufficient replicas, reducing fault tolerance.

[0004] Furthermore, when nodes fail or their state changes, the mapping relationship between the hash ring and nodes changes, requiring rapid and controllable location and replica repair of affected blocks. Existing solutions, lacking a unified mechanism for calculating storage location and handling state changes, are prone to issues such as untimely repair triggering, inconsistent repair target selection, and delayed metadata mapping updates. This can lead to block replicas remaining in a state of insufficient backups for extended periods, and even cause inconsistencies between read / write routes and actual storage locations. These problems are particularly pronounced in scenarios involving frequent node additions / exits, network fluctuations, and high-concurrency writes. Therefore, a block storage mechanism is urgently needed that can achieve capacity-weighted consistent hash location, controllable multi-replica write deduplication and completion, and rapid backup repair and metadata synchronization updates for failure scenarios within a group collaboration framework. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a blockchain collaborative storage method based on weighted consistent hashing. By electing anchor nodes within a logical collaboration group to uniformly maintain the consistent hash ring and metadata mapping relationship, a single-point coordinated block location and node change processing link is formed, reducing the risks of location inconsistencies and replica drift caused by distributed multi-point decision-making. By allocating the number of virtual nodes based on the available storage capacity of each physical node, the block landing point and replica distribution adaptively balance the storage capacity, alleviating overload of small-capacity nodes and idleness of large-capacity nodes. By generating a location hash value based on the new block identifier and backup sequence number, and searching clockwise on the hash ring combined with deduplication and completion rules, it is ensured that new block writing obtains n replicas distributed across different physical nodes, improving block availability and read / write continuity in single-node failure scenarios. By triggering ring updates through failure detection and performing incremental completion and mapping updates only on affected blocks, the amount of repair migration is reduced and recovery time is shortened.

[0006] To achieve the above objectives, the present invention provides a blockchain collaborative storage method based on weighted consistent hashing, comprising: Multiple blockchain physical nodes form a logical collaboration group. An anchor node is elected within the group, and the anchor node is responsible for calculating the block storage location and handling node state changes within the group. The available storage capacity of each physical node in the group is obtained from the anchor node. The number of virtual nodes is allocated to each physical node according to the available storage capacity. A set of virtual node coordinates is generated by using a cryptographic hash function based on the physical node identifier and the virtual node sequence number. The set of virtual node coordinates is sorted to form a consistent hash ring view. A mapping table from virtual node coordinates to physical nodes is established. The block storage backup number parameter n is determined by the anchor node. The backup number parameter n is used for new block writing and backup repair. When a new block is generated, a hash calculation is performed based on the new block identifier and backup sequence number to obtain n hash values ​​for locating in the consistent hash ring view. The target physical node is determined in the consistent hash ring view according to the clockwise search rule. After deduplication of the target physical nodes, if the number of physical nodes in the target physical node set is less than n, continue to search for the physical nodes mapped by the subsequent virtual nodes in a clockwise direction and fill them up to n different physical nodes. Distribute the new block data body to the target physical node set and record the metadata mapping relationship between the block identifier and the target physical node set after receiving the write confirmation. When the anchor node determines that a physical node has failed based on node status detection, the anchor node updates the consistent hash ring view and mapping table, filters out the affected blocks through the metadata mapping relationship, recalculates the target physical nodes for the affected blocks according to the updated consistent hash ring view, and determines the set of incremental nodes that need to be supplemented. The set of incremental nodes consists of nodes in the target physical node set other than the existing replica nodes. The data body of the affected block is obtained from the existing replica nodes and written to the incremental node set. After receiving the write confirmation, the metadata mapping relationship is updated so that the affected block satisfies the backup number parameter n and each backup is distributed on different physical nodes.

[0007] In the above technical solution, preferably, anchor nodes are elected within the group based on a consensus protocol; the anchor nodes have a preset term of office. When the continuous working time of the anchor node reaches the term duration, the anchor node triggers the rotation process to re-initiate the anchor node election to generate a new anchor node. The new anchor node issues a takeover completion notification to the group. From the effective date of the takeover completion notification, the new anchor node shall be responsible for calculating the block storage location and handling node status changes within the group. The original anchor node shall revert to a normal node after confirming the switchover is completed and continue to participate in block storage and replica maintenance.

[0008] In the above technical solution, preferably, the specific process of allocating the number of virtual nodes to each physical node based on the available storage capacity includes: Obtain the minimum available storage capacity of the physical nodes within the group, and configure a baseline number of virtual nodes for the minimum available storage capacity; For any physical node, the virtual node allocation value is determined based on the ratio of the available storage capacity of the physical node to the minimum available storage capacity. The virtual node allocation value is the rounded result of the product of the baseline virtual node number and the ratio.

[0009] In the above technical solution, preferably, the specific process of generating the virtual node coordinate set includes: For any physical node, the coordinates of multiple virtual nodes of the physical node are obtained by hashing the physical node identifier and the virtual node number assigned to that physical node using a cryptographic hash function. Sort the coordinates of all virtual nodes in the group according to their hash space coordinate values ​​to form the consistent hash ring view; Establish a mapping table from virtual node coordinates to physical nodes, wherein the mapping table uses the virtual node coordinates as the key and the physical node corresponding to the virtual node coordinates as the value.

[0010] In the above technical solution, preferably, the specific process of determining the block storage backup number parameter n by the anchor node includes: The stability probability of having at least one backup of a block is determined based on the failure probability of physical nodes within a preset time window, and the minimum stability probability threshold required for ledger continuity is set. As n increases from 1, a lower bound n of the basic number of backups is determined that makes the stability probability reach the minimum stability probability threshold. min ; The stability gain is calculated based on the stability gain threshold as the number of backups increases from n to n+1, and the smallest integer satisfying that the stability gain is less than the stability gain threshold is determined as the maximum number of backups n. max ; Configure the block storage backup number parameter n to satisfy n min ≤n≤n max Integers.

[0011] In the above technical solution, preferably, the specific process of obtaining the hash value for locating in the consistent hash ring view by hash calculation based on the new block identifier and backup sequence number includes: For each backup sequence number, the new block identifier and the corresponding backup sequence number are input into the cryptographic hash function for hash calculation to obtain n hash anchors; For any hash anchor, the corresponding target physical node is determined in the consistent hash ring view by following a clockwise search rule from the position of the hash anchor. The clockwise search rule is as follows: The first virtual node coordinate with a hash space coordinate value not less than the hash anchor point is searched. If it is not found, the virtual node coordinate with the smallest hash space coordinate value is looped back to the target physical node. The physical node mapped by the virtual node coordinate is then determined as the target physical node.

[0012] In the above technical solution, preferably, the specific process of writing the new block data body into the target physical node set includes: Distribute the new block data body to the target physical node set and receive write confirmation; When a write to a target physical node fails, the failed physical node is removed from the target physical node set. While maintaining the physical node deduplication constraint, the system continues to search for the physical node mapped by the subsequent virtual node clockwise along the consistent hash ring view. The found physical nodes are deduplicated and then added to the target physical node set until write confirmations are received from n different physical nodes.

[0013] In the above technical solution, preferably, the specific process by which the anchor node determines the failure of the physical node based on heartbeat detection includes: If any physical node in the group fails to return a heartbeat response within a preset number of consecutive heartbeat detection cycles, the physical node will be marked as invalid. The step of retrieving the data body of the affected block from the existing replica nodes and writing it into the incremental node set includes: Anchor nodes first retrieve the block cache on their side to obtain the data body of the affected block; When the block cache misses, a source node is selected from the existing replica nodes based on network load metrics and the data body of the affected block is retrieved. Write the data body of the affected block into the incremental node set and update the metadata mapping relationship after receiving the write confirmation.

[0014] In the above technical solution, preferably, the blockchain collaborative storage method based on weighted consistent hashing also includes a block rebalancing process after a new node joins, specifically including: When a new physical node is added to the group, the anchor node obtains the available storage capacity of the new physical node and determines the number of virtual nodes of the new physical node; Generate a set of virtual node coordinates for the new physical node and insert them into the consistent hash ring view to form an updated consistent hash ring view; For any newly inserted virtual node coordinates, the anchor node determines the hash range corresponding to the virtual node coordinates and filters the set of blocks to be migrated within the hash range, writes the block data body in the set of blocks to be migrated into the new physical node and receives the write confirmation; After the write confirmation, the anchor node sends a deletion command to the old physical node corresponding to the set of blocks to be migrated to perform post-cleanup. The deletion command takes effect when the backup number parameter n of the corresponding block is satisfied and the metadata mapping relationship is updated, and the metadata mapping relationship is updated.

[0015] This invention also discloses a blockchain collaborative storage system based on weighted consistent hashing, used to implement the blockchain collaborative storage method based on weighted consistent hashing disclosed in any of the above technical solutions, comprising: multiple blockchain physical nodes and an anchor node, wherein the anchor node is one of the multiple blockchain physical nodes; and including a virtual node allocation module, a consistent hashing ring construction module, a backup parameter management module, a new block location and writing module, a metadata mapping management module, a node status detection module, and a failure backup repair module configured corresponding to the anchor node; The anchor node is communicatively connected to the plurality of blockchain physical nodes, and the virtual node allocation module is used to obtain the available storage capacity of each physical node in the group and allocate the number of virtual nodes to each physical node according to the available storage capacity. The consistent hash ring construction module is used to generate a set of virtual node coordinates based on the physical node identifier and the virtual node sequence number using a cryptographic hash function, sort the set of virtual node coordinates to form a consistent hash ring view, and establish a mapping table from virtual node coordinates to physical nodes. The backup parameter management module is used to determine the block storage backup number parameter n, and provide the backup number parameter n to the new block positioning and writing module and the failed backup repair module; The new block location and write module is used to perform hash calculation based on the new block identifier and backup sequence number when a new block is generated to obtain a hash value for location in the consistent hash ring view, and determine the target physical node in the consistent hash ring view according to the clockwise search rule. After deduplication of the target physical node, if the number of physical nodes in the target physical node set is less than n, continue to search for the physical nodes mapped by the subsequent virtual nodes in a clockwise direction and fill them up to n different physical nodes. The new block data body is distributed to the target physical node set and write confirmation is received. The metadata mapping management module is used to record the metadata mapping relationship between block identifiers and target physical node sets, and to filter affected blocks based on the metadata mapping relationship after the consistent hash ring view is updated; The node status detection module is used to determine the failure of physical nodes based on node status detection and output a failure indication to the failure backup and repair module. The failure backup repair module is used to drive the consistent hash ring construction module to update the consistent hash ring view and the mapping table when a physical node fails, and to recalculate the target physical node for the affected block according to the updated consistent hash ring view and determine the incremental node set that needs to be supplemented. The incremental node set is the nodes in the target physical node set other than the existing replica nodes. The failure backup repair module is also used to obtain the data body of the affected block from the existing replica nodes and write it into the incremental node set, and update the metadata mapping relationship after the write confirmation, so that the affected block satisfies the backup number parameter n and each backup is distributed on different physical nodes.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By electing anchor nodes within the logical collaboration group and having them uniformly undertake the calculation of block storage location and the processing of node status changes, a consistent storage location decision and replica maintenance method within the group is achieved, reducing the risk of location inconsistency and replica drift caused by multi-point concurrent decision-making and improving the stability of group collaboration.

[0017] (2) By allocating the number of virtual nodes based on the available storage capacity of each physical node, and generating virtual node coordinates, constructing a consistent hash ring and mapping table using physical node identifiers and virtual node serial numbers, the weighted and balanced allocation of storage capacity by block landing points is realized, which alleviates the overload of small capacity nodes and the idleness of large capacity nodes, and improves the utilization rate of storage resources and the load balancing level.

[0018] (3) The backup availability stability probability is calculated by using failure statistics based on a preset time window, and the minimum number of backups parameter n is selected with the reliability target value. This achieves parameterized matching between the backup size and the reliability target, avoiding insufficient availability due to too small a number of backups or excessive storage and network overhead due to too large a number of backups.

[0019] (4) By generating n hash anchors based on the new block identifier and backup sequence number when the new block is generated, and by searching clockwise on the consistent hash ring and combining the physical node deduplication and completion rules to determine n different target physical nodes, the new block is written once to form a cross-node distributed replica, which improves the block availability and read / write continuity in the single node failure scenario.

[0020] (5) By removing failed write nodes during the writing process and searching for replacement nodes clockwise along the hash ring until write confirmations are obtained from n different physical nodes, self-recovery and rapid replacement of the write phase are achieved, reducing the impact of instantaneous abnormalities of local nodes on the write success rate and write latency.

[0021] (6) After determining the failure of a physical node based on heartbeat detection, the hash ring and mapping table are updated. The affected blocks are filtered according to the metadata mapping relationship, the target nodes of the affected blocks are recalculated and the incremental node set is determined. The data body is pulled from the existing replica nodes to complete the writing and update the mapping relationship. This realizes incremental repair in failure scenarios, reduces unnecessary data migration and shortens the recovery time, and enables the affected blocks to continuously meet the backup number parameter n and the cross-physical node distribution constraints.

[0022] (7) By generating virtual node coordinates and inserting them into the hash ring when a new node is added, filtering the blocks to be migrated according to the hash interval on the ring and writing them into the new node, and performing post-cleanup after the backup number is satisfied and the mapping is updated, controllable rebalancing after node expansion is achieved, reducing the overhead of full migration and improving the business continuity and storage balance effect of the expansion process. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the process framework of a blockchain collaborative storage method based on weighted consistent hashing disclosed in one embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, a blockchain collaborative storage method based on weighted consistent hashing according to the present invention includes: During the operation of a blockchain network, multiple physical blockchain nodes are grouped into a logical collaborative group. Each physical node elects an anchor node within the group, which acts as the unified scheduling entity within the group. The anchor node is centrally responsible for calculating the block storage location and handling node state changes within the group, thereby forming a consistent addressing and maintenance standard within the group and avoiding inconsistencies in mapping caused by multiple nodes calculating independently.

[0026] Anchor nodes periodically obtain the available storage capacity of each physical node within the group and allocate the number of virtual nodes to each physical node based on the available storage capacity, thus modeling the node's storage capacity as the number of virtual nodes. Specifically, anchor nodes generate a set of virtual node coordinates based on the physical node identifier and the virtual node sequence number using a cryptographic hash function. All virtual node coordinates within the group are sorted and concatenated to form a consistent hash ring view. Simultaneously, a mapping table is established between virtual node coordinates and physical nodes, mapping each physical node within the group to several virtual nodes on the hash ring. Using this weighted consistent hash topology, nodes with average storage resources are mapped to only a smaller number of virtual nodes, bearing only a corresponding proportion of the block storage load. Meanwhile, physical nodes with higher storage resources cover a larger area on the hash ring, thus bearing a higher proportion of the block storage load, reducing the ledger storage pressure on nodes with weaker resources, and effectively improving the load matching capability and operational stability in heterogeneous blockchain node groups.

[0027] The anchor node determines the block storage backup number parameter n, which is used for new block writing and backup repair. When a new block is generated, a hash calculation is performed based on the new block identifier and backup sequence number to obtain n hash values ​​for locating in the consistent hash ring view. The anchor node locates the target physical node in the consistent hash ring view according to the clockwise search rule and performs deduplication on the target physical node.

[0028] After deduplication of the target physical nodes, if the number of physical nodes in the target physical node set is less than n, the process continues clockwise to find the physical nodes mapped by the subsequent virtual nodes and supplements them to n different physical nodes. Subsequently, the anchor node distributes the new block data body to the target physical node set and receives write confirmation. After confirming that the write to n different physical nodes is satisfied, the anchor node records the metadata mapping relationship between the block identifier and the target physical node set for subsequent backup repair and migration location.

[0029] When the anchor node determines that a physical node has failed based on node status detection, it uses the anchor node to update the consistent hash ring view and mapping table, and filters out the affected blocks through the metadata mapping relationship.

[0030] For any affected block, recalculate the target physical nodes according to the updated consistent hash ring view and determine the set of incremental nodes that need to be supplemented. The set of incremental nodes consists of nodes in the target physical node set other than the existing replica nodes.

[0031] The anchor node retrieves the data body of the affected block from the existing replica nodes and writes it to the incremental node set. After receiving the write confirmation, it updates the metadata mapping relationship so that the affected block satisfies the backup number parameter n and each backup is distributed on different physical nodes.

[0032] In this implementation, by combining storage capacity-weighted virtual node mapping, consistent hashing with minimum remapping characteristics, and deduplication isolation with multiple replicas, deterministic block location and load distribution based on capacity are achieved in environments with heterogeneous nodes and dynamic node changes. In the event of node failure, backups are quickly replenished, ensuring ledger continuity and reducing the overhead of large-scale data migration.

[0033] In the above implementation, preferably, anchor nodes are elected within the group based on a consensus protocol; anchor nodes have a preset term of office to limit the centralization risk caused by a single node assuming the role of anchor node for a long period of time.

[0034] When an anchor node's continuous working time reaches its term limit, it triggers a rotation process to re-initiate an anchor node election to produce a new anchor node. The new anchor node issues a takeover completion notification to the group. From the effective date of the takeover completion notification, the new anchor node is responsible for calculating the block storage location within the group and handling node state changes. The original anchor node, after confirming the switchover is complete, reverts to a regular node and continues to participate in block storage and replica maintenance, ensuring that the rotation process does not interrupt the distribution of existing block replicas and write collaboration.

[0035] During implementation, let the set of physical nodes within the group be . Within the group, the Raft consensus algorithm is used to elect nodes as anchor nodes. In addition, to avoid a single node occupying the anchor node position for a long time, a fixed term of office is set for the anchor nodes. When the anchor node's continuous working time reaches the maximum term of office, the anchor node actively triggers the rotation process.

[0036] In this implementation, the anchor node management mechanism with term limits and rotation reduces the risk of single-point control caused by long-term fixed anchor nodes, and enables the addressing and repair responsibilities to be rotated controllably within the group, thereby improving the stability and decentralization of the group operation.

[0037] In the above embodiments, preferably, the specific process of allocating the number of virtual nodes to each physical node based on the available storage capacity includes: Get the minimum available storage capacity of physical nodes within the group. Configure a baseline number of virtual nodes for the minimum available storage capacity. ; For any physical node, calculate the ratio of its available storage capacity to its minimum available storage capacity. Multiply the base number of virtual nodes by this ratio to obtain the virtual node allocation value. Round this allocation value to the nearest integer and ensure that the number of virtual nodes is not less than 1, so that each physical node obtains a valid mapping point on the hash ring.

[0038] During implementation, for any physical node in the network Its actual usable storage capacity is Based on the principle of proportional allocation of storage resources, nodes Number of virtual nodes to be undertaken The calculation method is defined as follows: ; in, This reflects the storage capacity multiple of the node relative to the smallest node in the network, through comparison with... Multiplying these gives the number of virtual nodes that match the storage capacity of that node, rounded down to the nearest integer. This is used to ensure that the number of virtual nodes is an integer.

[0039] In this implementation, the minimum capacity node and the baseline number of virtual nodes are used as anchoring benchmarks, so that the number of virtual nodes increases proportionally with the storage capacity, thereby improving the load balancing accuracy and suppressing the overload of weak nodes, thus enhancing the stability of the group collaborative storage topology.

[0040] In the above embodiments, preferably, the specific process of generating the virtual node coordinate set includes: For any physical node, for each virtual node number assigned to that physical node, the physical node identifier and the virtual node number are respectively input into the cryptographic hash function to perform hash calculation to obtain the coordinates of multiple virtual nodes; Sort the coordinates of all virtual nodes in the group according to their hash space coordinate values ​​to form a consistent hash ring view.

[0041] Subsequently, a mapping table is established from virtual node coordinates to physical nodes. The mapping table uses the virtual node coordinates as the key and the physical node corresponding to the virtual node coordinates as the value, providing a unified data structure foundation for new block positioning and local remapping after node changes.

[0042] During implementation, physical nodes are determined. Number of virtual nodes Then, the anchor node maps the physical node to the hash ring using a hash algorithm. A discrete coordinate position. Let the physical node be... Its inherent unique feature is (Such as the node's IP address or MAC address), for that node, the first... virtual nodes ( Its position coordinates on the hash ring The calculation formula is: ; in This is a unique characteristic of physical nodes, used to ensure that the mapping intervals of different physical nodes do not interfere with each other. From 1 to an incrementing integer sequence number, This uses a standard cryptographic hash function (such as SHA-256). Through the above calculations, the physical node... Logically mapped to A set of virtual nodes distributed at different locations on the hash ring. The anchor node stores the coordinates of all virtual nodes. Sort the values ​​in ascending order and connect them end-to-end to form a hash ring view. Then, create a hash mapping table where virtual node locations are used as keys and physical nodes are used as values. This serves as a unified basic data structure for subsequent block location and migration calculations.

[0043] In this implementation, by generating discrete coordinates through multiple hashes, sorting all data into a ring, and mapping coordinates to nodes, the fine-grainedness of load distribution is improved without changing the addressing rules, and a deterministic basis is provided for subsequent clockwise search and local updates.

[0044] In the above embodiments, preferably, the specific process of determining the block storage backup number parameter n by the anchor node includes: Based on the failure probability or failure statistics of physical nodes within a preset time window, calculate the stability probability that at least one backup of a block is available, and set the minimum stability probability threshold required for ledger continuity.

[0045] As n increases from 1, determine the lower bound n of the basic number of backups that makes the stability probability reach the minimum stability probability threshold. min ; Furthermore, the anchor node calculates the stability gain as the number of backups increases from n to n+1 based on the stability gain threshold, sets the stability gain threshold, and determines the smallest integer that satisfies the stability gain being less than the stability gain threshold as the maximum number of backups n. max .

[0046] Finally, the block storage backup number parameter n is configured to satisfy n min ≤n≤n max The integer value ensures that the number of backups is subject to both reliability and marginal benefit constraints.

[0047] During implementation, to ensure continued availability of blocks in the event of node failure, the system sets the number of block storage backups as a configurable parameter. Based on a node failure probability model, a reasonable range of values ​​for this parameter is given. Let the probability that a single physical node fails within a given time window, rendering its storage backup unusable, be given. And it is assumed that failure events of different physical nodes are independent of each other. When a block is in When storing backups on multiple different physical nodes, the stable probability that at least one backup is available is: ; Based on the minimum stability probability threshold required for ledger continuity. ( To meet stability requirements, the minimum number of basic backups is set. Must meet: ; Therefore, we can conclude that: ; As the number of backups increases The stability gain increases monotonically and approaches 1, but its improvement exhibits diminishing marginal returns. When the number of backups is large, further increasing the number of backups provides limited stability gains but significantly increases storage usage and write overhead. Therefore, a stability gain threshold is introduced into the system. ( ), define the number of backups from Increase to The stability gain is: ; When the following conditions are met: ; At that point, the system determines that continuing to add backups has fallen below a perceptible threshold in terms of improving overall stability, and thus sets the minimum integer that satisfies the above inequality to [the specified value]. As the maximum number of backups: ; Based on the above, the system sets the backup number parameter. Must meet: The default setting is acceptable. To reduce storage overhead; when the system requires higher fault tolerance, it can be limited to no more than Improve under the premise In this way, the number of backups is determined by both stability constraints and marginal revenue thresholds, thus ensuring ledger continuity while also taking into account resource utilization efficiency.

[0048] In this implementation, the number of backups is controlled within a reasonable range that satisfies ledger continuity by using a dual threshold constraint of reliability lower limit and marginal benefit upper limit. The constraint screening mechanism ensures that block backups are distributed across different physical nodes, thereby reducing the risk of simultaneous failure of block backups due to failure and avoiding availability risks caused by too few backups and increased storage usage and write overhead caused by too many backups.

[0049] In the above embodiments, preferably, the specific process of obtaining the hash value for locating in the consistent hash ring view by hash calculation based on the new block identifier and backup sequence number includes: For each backup sequence number, the new block identifier and the corresponding backup sequence number are input into the cryptographic hash function for hash calculation, resulting in n hash anchors; each hash anchor corresponds to a positioning starting point in the consistent hash ring view.

[0050] For any hash anchor, the target physical node is determined in the consistent hash ring view by a clockwise search rule starting from the anchor position. The clockwise search rule is as follows: The system searches for the coordinates of the first virtual node whose hash space coordinates are not less than the hash anchor point. If not found, it wraps back to the coordinates of the virtual node with the smallest hash space coordinate value and determines the physical node mapped to the virtual node coordinates as the target physical node. By using multi-anchor point positioning, different backup nodes are distributed discretely on the hash ring, reducing the risk of strong coupling caused by multiple replicas falling into the same group of nodes.

[0051] In this implementation, by using multiple hash anchors, clockwise lookup, and wraparound rules, the discreteness of replica selection is improved while maintaining deterministic addressing of consistency hashes, and the risk of clustering in the same interval is reduced.

[0052] In the above embodiments, preferably, the specific process of writing the new block data body into the target physical node set includes: Distribute the new block data body to the target physical node set and receive write confirmation; When a write to a target physical node fails, the anchor node removes the failed physical node from the target physical node set. While maintaining the physical node deduplication constraint, it continues to search clockwise along the consistent hash ring view for the physical node mapped by the subsequent virtual node. After deduplication of the found physical node, it adds it to the target physical node set until write confirmations are received from n different physical nodes.

[0053] During implementation, the specific block writing algorithms include: Specifically, the new block identifier is set as (The new block hash or block header digest can be retrieved), its data body is Anchor node initializes the target physical node set The calculations are performed according to the above-described construction algorithm to finally obtain a set of storage nodes that satisfy physical isolation and have highly discrete characteristics. .

[0054] After obtaining the target physical node set Then, the anchor node will transfer the new block data body. Distribute to collection Each physical node performs the write operation, and each node returns an acknowledgment message upon successful write. When the anchor node receives a message from the set After receiving confirmation from all nodes, the new block is considered successfully written, and the metadata maintenance entity records the backup distribution mapping of the new block. ; in, This is a metadata mapping table representing blocks to a set of backup physical nodes, used to determine the backup source and target during subsequent node failure migration and new node addition migration.

[0055] If some target physical nodes fail to be written during the writing process, the anchor node continues to scan subsequent virtual nodes clockwise along the hash ring while maintaining the deduplication constraint of physical nodes, selecting new physical nodes to make up the backup number, until the requirement is met. And complete the confirmation to ensure that the new block write ultimately meets the preset backup number requirement.

[0056] In this implementation, by eliminating failures, filling in clockwise positions, and confirming deduplication, the system adaptively replaces nodes that are momentarily unavailable during the write phase, ensuring that the new blocks ultimately meet the preset backup number and maintain physically isolated distribution, thereby improving the write completion rate and replica reliability.

[0057] In the above implementation, preferably, when the anchor node determines that a physical node has failed based on heartbeat detection, periodic heartbeat detection is performed on any physical node within the group. The specific process includes: If any physical node in the group fails to return a heartbeat response within a preset number of consecutive heartbeat detection cycles, the physical node is marked as invalid, and the consistent hash ring view and mapping table are updated, as well as the affected blocks are filtered.

[0058] Retrieving the data body of the affected block from existing replica nodes and writing it to the incremental node set includes: Anchor nodes first retrieve the block cache on their side to obtain the data body of the affected block; When a block cache miss occurs, the source node is selected from the existing replica nodes based on network load metrics, and the data body of the affected block is pulled. Write the data body of the affected block into the incremental node set, and update the metadata mapping relationship after receiving the write confirmation, so that the repaired replica distribution satisfies the backup number parameter n and the physical isolation constraint.

[0059] Specifically, the anchor node records the mapping from the block to the set of backup physical nodes during the new block writing phase. Therefore, when a node failure occurs, the anchor node does not need to traverse the entire network; it can directly obtain the set of blocks to be repaired and migrated, along with their corresponding available backup source node information, from its local metadata. The specific execution flow of the data migration algorithm is as follows: During implementation, the anchor node runs a heartbeat-based keep-alive protocol. The number of consecutive packet losses exceeds the threshold Then, mark it as "invalid" and update the hash ring view. and mapping table And lock the set of affected blocks: .

[0060] For any affected block The anchor node calculates its position in the updated hash ring. The following should correspond to the set of target backup physical nodes. Define the set of surviving replicas after failure. : ; The set of incremental nodes that need to be supplemented for: ; During data recovery, the anchor node first searches its local cache; if a cache miss occurs, it searches the live set. The source node is selected based on network load metrics. Pull To local. Anchor node will Unified writing to incremental set To be assembled. After each node successfully writes data and returns confirmation information, the anchor node updates its metadata mapping as follows: ; Thus the block Even after a node fails, the preset backup number parameter is still met. And ensure that each backup is located on a different physical node.

[0061] In this implementation, by using continuous heartbeats to determine failure, prioritizing caching, and selectively fetching data based on network load, the fault detection and repair link is shortened, network congestion and recovery latency during the repair phase are reduced, and ledger availability and recovery efficiency are improved in failure scenarios.

[0062] In the above embodiments, preferably, the blockchain collaborative storage method based on weighted consistent hashing further includes a block rebalancing process after a new node joins, specifically including: When a new physical node is added to the group, the anchor node obtains the available storage capacity of the new physical node and determines the number of virtual nodes of the new physical node; it generates a set of virtual node coordinates of the new physical node and inserts them into the consistent hash ring view to form an updated consistent hash ring view.

[0063] For any newly inserted virtual node coordinates, the anchor node determines the hash range corresponding to the virtual node coordinates, filters the set of blocks to be migrated within the hash range, writes the block data body of the set of blocks to be migrated into the new physical node, and receives the write confirmation.

[0064] After the write confirmation, the anchor node sends a deletion command to the old physical node corresponding to the set of blocks to be migrated to perform post-cleanup. The deletion command takes effect when the backup number parameter n of the corresponding block is satisfied and the metadata mapping relationship is updated, thereby releasing the storage occupied by the old node and maintaining the consistency of the mapping relationship after the migration is completed.

[0065] Specifically, when a new physical node is added to the group for expansion... At this time, the anchor node is responsible for smoothly migrating part of the storage load from the existing node to the new node while maintaining hash consistency monotonicity. The node joining process specifically includes: During implementation, the anchor node assesses its storage capacity. Based on this, the number of virtual nodes to be allocated is calculated. Anchor node generation A set of virtual node coordinates Insert it into the current hash ring to form the updated hash ring view. and the corresponding virtual-physical mapping table .

[0066] For any newly inserted virtual node position Anchor nodes are filtered within the interval Blocks within Hash calculations are performed on the blocks in this range to generate a set to be migrated. : ; ; ; The anchor node iterates through the elements in the collection and initiates a block write task. A post-cleanup strategy is executed, and confirmation is then performed. After a successful write, the anchor node sends a delete command to the old nodes belonging to the target set.

[0067] In this implementation, the minimum remapping characteristic of consistent hashing is used to limit the impact of expansion to the local interval corresponding to the new virtual node, so that only a small number of affected blocks have their storage mapping relationships adjusted, thereby avoiding large-scale data migration; through write confirmation followed by cleanup, rebalancing is smoothly completed while ensuring the consistency of replica count and metadata, reducing the large-scale migration disturbance caused by expansion.

[0068] This invention also discloses a blockchain collaborative storage system based on weighted consistent hashing, used to implement the blockchain collaborative storage method based on weighted consistent hashing disclosed in any of the above embodiments. The system includes: multiple blockchain physical nodes and an anchor node, where the anchor node is one of the multiple blockchain physical nodes and maintains communication connections with other physical nodes in the group; and includes a virtual node allocation module, a consistent hashing ring construction module, a backup parameter management module, a new block location and writing module, a metadata mapping management module, a node status detection module, and a failure backup and repair module, all configured corresponding to the anchor node. Each module is deployed on the anchor node side as a process or service and interacts with the block storage service on the physical node side.

[0069] Anchor nodes communicate with multiple blockchain physical nodes. The virtual node allocation module is used to obtain the available storage capacity of each physical node in the group and allocate the number of virtual nodes to each physical node according to the available storage capacity. The Consistent Hash Ring Construction Module is used to generate a set of virtual node coordinates based on the physical node identifier and the virtual node sequence number using a cryptographic hash function, sort the set of virtual node coordinates to form a consistent hash ring view, and establish a mapping table from virtual node coordinates to physical nodes. The backup parameter management module is used to determine the number of block storage backups, n, and provide the number of backups, n, to the new block location writing module and the failed backup repair module. The new block location and write module is used to perform hash calculation based on the new block identifier and backup sequence number when a new block is generated to obtain a hash value for location in the consistent hash ring view. It then determines the target physical node in the consistent hash ring view according to the clockwise search rule. After deduplication of the target physical node, if the number of physical nodes in the target physical node set is less than n, it continues to search for the physical nodes mapped by the subsequent virtual nodes in a clockwise direction and fills in n different physical nodes. Finally, it distributes the new block data body to the target physical node set and receives write confirmation. The metadata mapping management module is used to record the metadata mapping relationship between block identifiers and target physical node sets, and to filter affected blocks based on the metadata mapping relationship after the consistent hash ring view is updated; The node status detection module is used to determine the failure of physical nodes based on node status detection and output a failure indication to the failure backup and repair module; The failure backup and repair module is used to drive the consistent hash ring construction module to update the consistent hash ring view and mapping table when a physical node fails. It also recalculates the target physical node for the affected block according to the updated consistent hash ring view and determines the incremental node set that needs to be supplemented. The incremental node set consists of nodes in the target physical node set other than the existing replica nodes. The failure backup repair module is also used to obtain the data body of the affected block from the existing replica nodes and write it into the incremental node set. After the write is confirmed, the metadata mapping relationship is updated so that the affected block satisfies the backup number parameter n and each backup is distributed on different physical nodes.

[0070] The blockchain collaborative storage system based on weighted consistent hashing disclosed in the above embodiments has modules whose functions correspond to the steps of the blockchain collaborative storage method based on weighted consistent hashing disclosed in the above embodiments. In the implementation process, the above embodiments are referred to for operation, and will not be repeated here.

[0071] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A blockchain collaborative storage method based on weighted consistent hashing, characterized in that, include: Multiple blockchain physical nodes form a logical collaboration group. An anchor node is elected within the group, and the anchor node is responsible for calculating the block storage location and handling node state changes within the group. The available storage capacity of each physical node in the group is obtained from the anchor node. The number of virtual nodes is allocated to each physical node according to the available storage capacity. A set of virtual node coordinates is generated by using a cryptographic hash function based on the physical node identifier and the virtual node sequence number. The set of virtual node coordinates is sorted to form a consistent hash ring view. A mapping table from virtual node coordinates to physical nodes is established. The block storage backup number parameter n is determined by the anchor node. The backup number parameter n is used for new block writing and backup repair. When a new block is generated, a hash calculation is performed based on the new block identifier and backup sequence number to obtain n hash values ​​for locating in the consistent hash ring view. The target physical node is determined in the consistent hash ring view according to the clockwise search rule. After deduplication of the target physical nodes, if the number of physical nodes in the target physical node set is less than n, continue to search for the physical nodes mapped by the subsequent virtual nodes in a clockwise direction and fill them up to n different physical nodes. Distribute the new block data body to the target physical node set and record the metadata mapping relationship between the block identifier and the target physical node set after receiving the write confirmation. When the anchor node determines that a physical node has failed based on node status detection, the anchor node updates the consistent hash ring view and mapping table, filters out the affected blocks through the metadata mapping relationship, recalculates the target physical nodes for the affected blocks according to the updated consistent hash ring view, and determines the set of incremental nodes that need to be supplemented. The set of incremental nodes consists of nodes in the target physical node set other than the existing replica nodes. The data body of the affected block is obtained from the existing replica nodes and written to the incremental node set. After receiving the write confirmation, the metadata mapping relationship is updated so that the affected block satisfies the backup number parameter n and each backup is distributed on different physical nodes.

2. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1, characterized in that, Anchor nodes are elected within the group based on a consensus protocol. The anchor node has a preset term of office; When the continuous working time of the anchor node reaches the term duration, the anchor node triggers the rotation process to re-initiate the anchor node election to generate a new anchor node. The new anchor node issues a takeover completion notification to the group. From the effective date of the takeover completion notification, the new anchor node shall be responsible for calculating the block storage location and handling node status changes within the group. The original anchor node shall revert to a normal node after confirming the switchover is completed and continue to participate in block storage and replica maintenance.

3. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1, characterized in that, The specific process of allocating the number of virtual nodes to each physical node based on the available storage capacity includes: Obtain the minimum available storage capacity of the physical nodes within the group, and configure a baseline number of virtual nodes for the minimum available storage capacity; For any physical node, the virtual node allocation value is determined based on the ratio of the available storage capacity of the physical node to the minimum available storage capacity. The virtual node allocation value is the rounded result of the product of the baseline virtual node number and the ratio.

4. The blockchain collaborative storage method based on weighted consistent hashing according to claim 3, characterized in that, The specific process of generating the set of virtual node coordinates includes: For any physical node, the coordinates of multiple virtual nodes of the physical node are obtained by hashing the physical node identifier and the virtual node number assigned to that physical node using a cryptographic hash function. Sort the coordinates of all virtual nodes in the group according to their hash space coordinate values ​​to form the consistent hash ring view; Establish a mapping table from virtual node coordinates to physical nodes, wherein the mapping table uses the virtual node coordinates as the key and the physical node corresponding to the virtual node coordinates as the value.

5. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1, characterized in that, The specific process of determining the block storage backup number parameter n by the anchor node includes: The stability probability of having at least one backup of a block is determined based on the failure probability of physical nodes within a preset time window, and the minimum stability probability threshold required for ledger continuity is set. As n increases from 1, a lower bound n of the basic number of backups is determined that makes the stability probability reach the minimum stability probability threshold. min ; The stability gain is calculated based on the stability gain threshold as the number of backups increases from n to n+1, and the smallest integer satisfying that the stability gain is less than the stability gain threshold is determined as the maximum number of backups n. max ; Configure the block storage backup number parameter n to satisfy n min ≤n≤n max Integers.

6. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1, characterized in that, The specific process of calculating the hash value used for locating in the consistent hash ring view based on the new block identifier and backup sequence number includes: For each backup sequence number, the new block identifier and the corresponding backup sequence number are input into the cryptographic hash function for hash calculation to obtain n hash anchors; For any hash anchor, the corresponding target physical node is determined in the consistent hash ring view by following a clockwise search rule from the position of the hash anchor. The clockwise search rule is as follows: The first virtual node coordinate with a hash space coordinate value not less than the hash anchor point is searched. If it is not found, the virtual node coordinate with the smallest hash space coordinate value is looped back to the target physical node. The physical node mapped by the virtual node coordinate is then determined as the target physical node.

7. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1 or 6, characterized in that, The specific process of writing the new block data body into the target physical node set includes: Distribute the new block data body to the target physical node set and receive write confirmation; When a write to a target physical node fails, the failed physical node is removed from the target physical node set. While maintaining the physical node deduplication constraint, the system continues to search for the physical node mapped by the subsequent virtual node clockwise along the consistent hash ring view. The found physical nodes are deduplicated and then added to the target physical node set until write confirmations are received from n different physical nodes.

8. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1, characterized in that, The specific process by which the anchor node determines the failure of a physical node based on heartbeat detection includes: If any physical node in the group fails to return a heartbeat response within a preset number of consecutive heartbeat detection cycles, the physical node will be marked as invalid. The step of retrieving the data body of the affected block from the existing replica nodes and writing it into the incremental node set includes: Anchor nodes first retrieve the block cache on their side to obtain the data body of the affected block; When the block cache misses, a source node is selected from the existing replica nodes based on network load metrics and the data body of the affected block is retrieved. Write the data body of the affected block into the incremental node set and update the metadata mapping relationship after receiving the write confirmation.

9. The blockchain collaborative storage method based on weighted consistent hashing according to claim 1, characterized in that, It also includes the block rebalancing process after a new node joins, specifically including: When a new physical node is added to the group, the anchor node obtains the available storage capacity of the new physical node and determines the number of virtual nodes for the new physical node; Generate a set of virtual node coordinates for the new physical node and insert them into the consistent hash ring view to form an updated consistent hash ring view; For any newly inserted virtual node coordinates, the anchor node determines the hash range corresponding to the virtual node coordinates and filters the set of blocks to be migrated within the hash range, writes the block data body in the set of blocks to be migrated into the new physical node and receives the write confirmation; After the write confirmation, the anchor node sends a deletion command to the old physical node corresponding to the set of blocks to be migrated to perform post-cleanup. The deletion command takes effect when the backup number parameter n of the corresponding block is satisfied and the metadata mapping relationship is updated, and the metadata mapping relationship is updated.

10. A blockchain collaborative storage system based on weighted consistent hashing, characterized in that, A blockchain collaborative storage method based on weighted consistent hashing as described in any one of claims 1 to 9 is provided, comprising: multiple blockchain physical nodes and an anchor node, wherein the anchor node is one of the multiple blockchain physical nodes; and includes a virtual node allocation module, a consistent hashing ring construction module, a backup parameter management module, a new block location and writing module, a metadata mapping management module, a node status detection module, and a failure backup repair module configured corresponding to the anchor node. The anchor node is communicatively connected to the plurality of blockchain physical nodes, and the virtual node allocation module is used to obtain the available storage capacity of each physical node in the group and allocate the number of virtual nodes to each physical node according to the available storage capacity. The consistent hash ring construction module is used to generate a set of virtual node coordinates based on the physical node identifier and the virtual node sequence number using a cryptographic hash function, sort the set of virtual node coordinates to form a consistent hash ring view, and establish a mapping table from virtual node coordinates to physical nodes. The backup parameter management module is used to determine the block storage backup number parameter n, and provide the backup number parameter n to the new block positioning and writing module and the failed backup repair module; The new block location and write module is used to perform hash calculation based on the new block identifier and backup sequence number when a new block is generated to obtain a hash value for location in the consistent hash ring view, and determine the target physical node in the consistent hash ring view according to the clockwise search rule. After deduplication of the target physical node, if the number of physical nodes in the target physical node set is less than n, continue to search for the physical nodes mapped by the subsequent virtual nodes in a clockwise direction and fill them up to n different physical nodes. The new block data body is distributed to the target physical node set and write confirmation is received. The metadata mapping management module is used to record the metadata mapping relationship between block identifiers and target physical node sets, and to filter affected blocks based on the metadata mapping relationship after the consistent hash ring view is updated; The node status detection module is used to determine the failure of physical nodes based on node status detection and output a failure indication to the failure backup and repair module. The failure backup repair module is used to drive the consistent hash ring construction module to update the consistent hash ring view and the mapping table when a physical node fails, and to recalculate the target physical node for the affected block according to the updated consistent hash ring view and determine the incremental node set that needs to be supplemented. The incremental node set is the nodes in the target physical node set other than the existing replica nodes. The failure backup repair module is also used to obtain the data body of the affected block from the existing replica nodes and write it into the incremental node set, and update the metadata mapping relationship after the write confirmation, so that the affected block satisfies the backup number parameter n and each backup is distributed on different physical nodes.

Images