State compaction consistent hashing method supporting arbitrary removal of evidence storage nodes

Abstract

The application provides a state compression consistent hashing method supporting arbitrary removal of an evidence storage node, comprising: constructing a state compression layer for recording a removal state of the storage node; when a storage node is removed, updating the removal state through the state compression layer with constant time complexity; based on the removal state recorded by the state compression layer, addressing a data key, and based on a stateless consistent hashing engine, performing primary mapping on the data key to obtain a first node number; when the storage node corresponding to the first node number is in a working state, outputting the first node number as an addressing result; or when the storage node corresponding to the first node number is in a removal state, triggering secondary mapping, recalculating the first node number in a current working storage node set, and outputting a second node number as an addressing result. The method realizes high efficient search performance and low memory overhead while supporting arbitrary random removal of the storage node.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a state compression consistent hashing method that supports arbitrary removal of evidence storage nodes. Background Technology

[0002] In distributed storage systems, especially in scenarios involving the storage of digital evidence (such as audit logs and surveillance videos), an efficient mapping method is needed to quickly and stably address data to specific storage nodes (or "buckets"). Traditional hash modulo methods trigger near-full data remapping when the number of nodes changes, leading to unbearable data migration and index rebuilding costs. Therefore, consistent hashing is widely used.

[0003] Existing consistent hashing methods are mainly divided into two categories: The first category is stateless consistent hashing, such as Jump consistent hashing and its variants PowerCH and FlipHash. These methods have excellent lookup performance and extremely low memory consumption, but their algorithm design is usually based on the assumption of bucket sequential numbering, which cannot support the random removal of storage nodes. The second category is consistent hashing that supports random node removal, such as consistent hashing rings based on virtual nodes, AnchorHash, and MementoHash. These methods can handle random node removal, but as the scale of storage nodes and the proportion of removed nodes increase, they often face problems such as extended lookup paths, severe performance degradation, or bloated state structures.

[0004] Therefore, how to provide a consistent hashing method that can maintain high search performance and low memory overhead while supporting arbitrary and random removal of storage nodes has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides a state compression consistent hashing method that supports arbitrary removal of evidence storage nodes, thereby solving the shortcomings of existing consistent hashing methods that cannot maintain high search performance and low memory overhead while supporting arbitrary random removal of storage nodes.

[0006] This invention provides a state compression consistent hashing method that supports arbitrary removal of evidence storage nodes, comprising: A state compression layer is constructed to record the removal status of storage nodes. The state compression layer includes a reciprocal array and a rollback stack. When a storage node is removed, the removal status is updated in constant time complexity through the state compression layer. Based on the removal status recorded by the state compression layer, the data key is addressed. A stateless consistent hash engine is used to perform a primary mapping on the data key to obtain a first node number. When the storage node corresponding to the first node number is in a working state, the first node number is output as the addressing result. Alternatively, when the storage node corresponding to the first node number is in a removed state, a secondary mapping is triggered. Based on the first node number, a second node number is recalculated in the currently working storage node set, and the second node number is output as the addressing result. The storage node corresponding to the second node number is in a working state.

[0007] According to the state compression consistent hashing method for supporting arbitrary removal of evidence storage nodes provided by the present invention, after outputting the first node number as the addressing result, the method further includes: If the state corresponding to the node with the first node number stored in the state compression layer is determined to be a working state, the search is terminated and the first node number is output; or... If it is determined that the state corresponding to the node with the first node number stored in the state compression layer is a removed state, then a secondary mapping is triggered.

[0008] According to the present invention, a state-compressed consistent hashing method supporting arbitrary removal of evidence storage nodes is provided, wherein the step of recalculating based on the first node number in the current set of working nodes and outputting the second node number as the addressing result includes: The first node number is used as input, and the intermediate hash value is calculated using a preset hash function; The intermediate hash value is moduloed by the total number of currently valid working nodes to obtain the second node number as the addressing result.

[0009] According to the present invention, a state compression consistent hashing method supporting arbitrary removal of evidence storage nodes is provided, wherein constructing a state compression layer includes: Create a first array to store the numbers of the storage nodes in index order; Create a second array that is the inverse of the first array, and use it to store the index position of the number of each storage node in the first array.

[0010] A state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to the present invention further includes: Based on the total number N of initial storage nodes, assign a storage node number value to each index position i of the first array, such that the first array [i] = i, where i ∈ [0, N); Assign an index position value to each storage node number i of the second array, such that the second array [i] = i, and establish an initial inverse mapping relationship between node numbers and array indices.

[0011] A state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to the present invention further includes: A rollback stack is provided to record array indices of content replacements caused by storage node removal operations. When a node needs to be restored, the array indices are popped from the rollback stack to guide the storage node to add the reverse operation.

[0012] According to the present invention, a state compression consistent hashing method for supporting arbitrary removal of evidence storage nodes is provided. When updating the removal state, the removal of the storage node includes: locating, replacing with the end slot of the working interval, shrinking the working interval and recording reversible information; The location includes: querying the second array based on the node number to be removed, and obtaining the target index of the node number to be removed in the first array; The replacement with the end slot of the work interval includes: modifying the value of the node number to be removed in the first array to the end index of the work interval, swapping the value at the target index in the first array with the value at the end index of the work interval, and synchronously updating the corresponding mapping relationship in the second array; The shrinking of the working interval and recording of reversible information includes: reducing the range of the working interval by one and pushing the target index onto the rollback stack.

[0013] According to the state compression consistent hashing method for supporting arbitrary removal of evidence storage nodes provided by the present invention, before performing the process of shrinking the working interval and recording reversible information, the method further includes: The process involves determining that the target index is equal to the end index of the working interval, marking the value at the target index in the first array as removed, and then performing the shrinking of the working interval and recording reversible information.

[0014] A state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to the present invention further includes: Pop the index recorded for the most recent removal operation from the top of the rollback stack; Based on the index recorded in the most recent removal operation, by performing steps that are the reverse of the removal operation logic, the storage node corresponding to the index recorded in the most recent removal operation is restored to the working state, and the working range is expanded.

[0015] A state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to the present invention further includes: Determine that the current number of working nodes equals the total number of storage nodes, add the newly added storage node to the end of the storage node list, and increment the total number of storage nodes count.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the state compression consistent hashing method for supporting arbitrary removal of evidence storage nodes as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the state-compressed consistent hashing method as described above that supports arbitrary removal of evidence storage nodes.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the state compression consistent hashing method as described above that supports arbitrary removal of evidence storage nodes.

[0019] This invention provides a state-compressed consistent hashing method that supports arbitrary removal of evidence storage nodes. It utilizes a stateless consistent hashing engine as the primary mapping and triggers a secondary mapping lookup mechanism only when mapping to a removed node. A state compression layer is constructed to compress and optimize the lookup path after a node removal. This method supports arbitrary node removal while maintaining high lookup performance, and also supports the removal of random storage nodes with low memory consumption. The state compression layer, constructed using a reciprocal array and a rollback stack, records the node removal state. Memory consumption is linearly related to the total number of nodes and is independent of the removal ratio, ensuring that state management overhead remains controllable and low. This method effectively supports arbitrary random removal of storage nodes while maintaining high lookup performance and low memory overhead. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the state compression consistent hashing method that supports arbitrary removal of evidence storage nodes provided by the present invention. Figure 2 This is a schematic diagram illustrating the principle of the state compression consistent hashing method that supports arbitrary removal of evidence storage nodes provided by the present invention; Figure 3 This is a schematic diagram of the main mapping hitting the working bucket in the search algorithm provided by this invention; Figure 4 This is a schematic diagram illustrating the two-stage search performed by the search algorithm provided in this invention; Figure 5 This is an initialization diagram provided by the present invention; Figure 6 This is a schematic diagram of array update after removing storage nodes in last-in-first-out order, provided by the present invention. Figure 7 This is a schematic diagram of the newly added storage node provided by the present invention; Figure 8 This is a schematic diagram of the traditional replacement set record removing the bucket status; Figure 9 This is a schematic diagram of the traditional replacement set search process; Figure 10 This is a schematic diagram of the compressed state of the array after randomly removing buckets, provided by the present invention. Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a flowchart illustrating the state compression consistent hashing method for supporting arbitrary removal of evidence storage nodes provided by the present invention. Figure 2 This is a schematic diagram illustrating the principle of the state compression consistent hashing method that supports arbitrary removal of evidence storage nodes provided by the present invention.

[0024] like Figure 1 and Figure 2 As shown, this invention provides a state compression consistent hashing method that supports arbitrary removal of evidence storage nodes. It achieves a balance between support for arbitrary removal of storage nodes and high performance through a state compression layer and two-phase addressing. It should be noted that removal and invalidation are synonymous; the academic term is bucket removal, while in practical business scenarios it is referred to as storage node failure or unavailability. The method mainly includes the following steps: 101. Construct a state compression layer to record the removal state of storage nodes. The state compression layer includes a reciprocal array and a rollback stack.

[0025] Specifically, the algorithm for removing storage nodes is the core of the state compression layer. The state compression layer uses two inverse arrays and a rollback stack to compress, store, and record the state of removed storage nodes. First, the state compression layer is constructed, consisting of two inverse arrays (the first array, `bucket_at`, and the second array, `pos_of`) and a rollback stack (`rb_stack`). Its core function is to compactly record the removal state of storage nodes (hereinafter referred to as "buckets"), avoiding the state bloat problem of traditional methods. The two inverse arrays are used to establish the mapping relationship between bucket numbers and array indices, and the rollback stack (`rb_stack`) is used to record key information about the removal operation, providing a basis for node recovery. The memory usage of the entire state compression layer is linearly related only to the total number of buckets and is not affected by the proportion of removed nodes.

[0026] 102. When a storage node is removed, the removal state is updated in constant time complexity through the state compression layer.

[0027] Specifically, when any bucket is removed or needs to be actively removed, the removal state is updated in constant time complexity of O(1) through the state compression layer. The constant time update mechanism avoids complex calculations during node removal and ensures fast response when nodes change dynamically.

[0028] 103. Based on the removal state recorded by the state compression layer, address the data key. Here, based on the stateless consistent hash engine, perform primary mapping on the data key to obtain the first node number. When the storage node corresponding to the first node number is in a working state, output the first node number as the addressing result. Alternatively, when the storage node corresponding to the first node number is in a removed state, trigger secondary mapping, recalculate based on the first node number in the currently working storage node set, and output the second node number as the addressing result. The storage node corresponding to the second node number is in a working state.

[0029] Specifically, such as Figure 2 As shown, addressing is performed according to a two-phase lookup mechanism. The first step is the main mapping, which calls a stateless consistent hash engine (such as PowerCH) to map the data key, obtaining the first node number. The stateless consistent hash engine boasts excellent addressing performance and low memory consumption, ensuring efficient and fast main mapping.

[0030] Query the removal status of the compressed layer record, when the first node number ( If the bucket corresponding to the first node is a working node (not removed), then directly output the number of that first node. ) as the addressing result. When the first node number ( If the bucket corresponding to is removed, a secondary mapping is triggered, based on the first node number ( The second node number is then recalculated within the current set of working nodes, and finally output as the addressing result.

[0031] This embodiment combines a state compression layer with a two-stage lookup, which supports the random removal of any bucket while maintaining the high-efficiency lookup performance of stateless consistent hashing, and controls memory overhead, avoiding large-scale data migration and index rebuilding.

[0032] Furthermore, based on the above embodiments, after outputting the first node number as the addressing result in this embodiment, it further includes: if the state corresponding to the node stored in the state compression layer is determined to be the working state, then the search is terminated and the first node number is output; or if the state corresponding to the node stored in the state compression layer is determined to be the removed state, then a secondary mapping is triggered.

[0033] Specifically, output the first node number ( After the addressing result is obtained, a second state verification is performed. First, the first node number is queried through the state compression layer ( The status identifier is recorded synchronously by the values ​​in the inverse arrays (the first array (bucket_at array) and the second array (pos_of array)). The working node is a valid number, and the removed node is a preset invalid value.

[0034] If the query result indicates a working status, the search process will terminate immediately, and the first node number will be output. This avoids redundant calculations. If the query result indicates a removed state, a secondary mapping process is immediately triggered to ensure that the addressing result always points to a valid working node.

[0035] By adding a state judgment step, the validity of the addressing result is further guaranteed, avoiding addressing errors caused by state synchronization delays, without increasing additional performance overhead, thus meeting low latency requirements. Combined with... Figure 3 As can be seen from the algorithm's finding that when the main mapping hits the working bucket, the result is directly output after the state is judged, making the process simple and efficient.

[0036] Furthermore, based on the above embodiments, this embodiment recalculates the second node number in the current set of working nodes based on the first node number and outputs the second node number as the addressing result, including: taking the first node number as input and calculating the intermediate hash value through a preset hash function; performing a modulo operation on the intermediate hash value with the total number of currently valid working nodes to obtain the second node number as the addressing result.

[0037] Specifically, to ensure that the keys corresponding to removed nodes are evenly distributed among worker nodes, when a secondary mapping is triggered, the following steps are performed to recalculate: The obtained first node number ( As the sole input parameter, this parameter inherently carries the correlation of the original mapping, ensuring the balance of the allocation.

[0038] The first node is numbered by calling the preset hash function. Hash operations are performed, and in this embodiment, the FNV-1a-64 function from the Fowler–Noll–Vo (FNV) hash function family with 64-bit output is preferred. This function has the characteristics of uniform distribution and fast calculation, and can avoid uneven distribution caused by hash collisions.

[0039] The intermediate hash value is then moduloed by the total number of currently valid worker nodes (W), i.e., target index (idx) = Hash(k, The second node number is obtained by calculating the current number of valid working nodes (W) modulo the target index (idx) of the operation result, and the bucket number corresponding to the target index (idx) is the second node number. The second node number is then output as the addressing result.

[0040] By explicitly defining the hash function type and modulo logic, it is ensured that the keys after the secondary mapping are evenly distributed across the worker node set. This achieves the effect of keeping the key mapping unchanged when the primary mapping bucket is available and minimizing disturbance when the primary mapping bucket is removed, ensuring even key migration. Figure 4 The search algorithm performs a two-stage search, which can quickly locate the effective working bucket and avoid lengthening the query path.

[0041] Furthermore, based on the above embodiments, this embodiment constructs a state compression layer, including: creating a first array for storing the numbers of storage nodes in index order; and creating a second array that is the inverse of the first array for storing the index position of the number of each storage node in the first array.

[0042] Specifically, building the state compression layer mainly involves creating two arrays with inverse functions, which work together to achieve a fast mapping between bucket numbers and index positions.

[0043] The first array, also known as the first array (bucket_at array), stores the bucket numbers in index order. The index position of the array is associated with the physical storage location of the bucket. The corresponding bucket number can be directly obtained through the index. For example, the bucket number corresponding to index i is the first array (bucket_at array) [i].

[0044] The second array, or pos_of array, is the inverse of the first array (bucket_at array). It stores the index position of each bucket number in the first array (bucket_at array). That is, you can directly query its index in the first array (bucket_at array) through the bucket number (pos). Target index (idx) = second array (pos_of array) [bucket number (pos)].

[0045] Two inverse arrays enable queries for both bucket numbers and index positions to be completed in O(1) time, laying the foundation for efficient execution of node removal, addition, and search operations. Figure 5 As shown in the initialization diagram, the mapping relationship between the two arrays is clear after initialization, which facilitates dynamic adjustment.

[0046] Furthermore, based on the above embodiments, this embodiment also includes: assigning a storage node number value to each index position i of the first array according to the total number N of the initial storage nodes, such that the first array [i] = i, where i ∈ [0, N); assigning an index position value to each storage node number i of the second array, such that the second array [i] = i, and establishing an initial inverse mapping relationship between the node number and the array index.

[0047] Specifically, to clarify the initialization process of the two inverse arrays and ensure the accuracy of the initial mapping relationship, the total number N of initial storage nodes, i.e., the total number of initial buckets, is obtained, and the two arrays are initialized accordingly. Assign a value to each index position i (where i ranges from [0, N)) of the first array (bucket_at array) so that the first array (bucket_at array)[i] = i, that is, the index position is completely consistent with the corresponding bucket number in the initial state.

[0048] Assign a value to each storage node number i (the value range of i is also [0,N)) in the second array (pos_of array) so that the second array (pos_of array)[i]=i, that is, the bucket number is completely consistent with its index position in the first array (bucket_at array) in the initial state.

[0049] By initializing the values, an initial inverse mapping relationship is established between the bucket number and the array index. At this point, the total number W of currently valid work nodes in the work interval is equal to the total number N of the initial storage nodes. Figure 5 As shown, when the total number of initial storage nodes N=7, W=N=7, and pos∈[0,7-1], all index positions of the first array (bucket_at array) and the second array (pos_of array) are assigned to themselves. The algorithm maintains the working mode of the PowerCH consistent hash function to ensure efficient addressing in the initial state.

[0050] Furthermore, based on the above embodiments, this embodiment also includes: providing a rollback stack, recording the array index of content replacement caused by the storage node removal operation, which is used to pop the array index from the rollback stack to guide the storage node to add the reverse operation when the node needs to be restored.

[0051] Specifically, the construction and implementation of the rollback stack (rb_stack) provide support for the reversible recovery of storage nodes. A rollback stack (rb_stack) is added to the state compression layer. The core function of the rollback stack (rb_stack) is to record the index of the first array (bucket_at array) whose content is replaced due to bucket removal operations: whenever a bucket removal operation is performed and involves content replacement at the index position in the first array (bucket_at array), the affected index position is pushed onto the rollback stack (rb_stack); when it is necessary to recover the removed bucket, the most recently recorded index position is popped from the top of the rollback stack (rb_stack). This index position directly guides the execution of the recovery operation, ensuring that the recovery process is the reverse of the removal operation and avoiding mapping confusion.

[0052] The rollback stack (rb_stack) setting ensures that bucket addition / restoration operations do not require rebuilding the entire mapping relationship; only the inverse operation needs to be performed based on the index recorded in the stack. This guarantees the efficiency of addition operations. Figure 6 Array update status after removal and Figure 7 As can be seen from the newly added storage node, the rollback stack (rb_stack) can accurately record the removal traces, ensuring that the array state is consistent with that before removal after restoration.

[0053] Furthermore, based on the above embodiments, this embodiment removes storage nodes when updating the removal status, including: locating, replacing with the end slot of the working interval, shrinking the working interval, and recording reversible information. Locating includes: querying the second array based on the node number to be removed, and obtaining the target index of the node number to be removed in the first array; replacing with the end slot of the working interval includes: modifying the value of the node number to be removed in the first array to the end index of the working interval, swapping the value at the target index in the first array with the value at the end index of the working interval, and synchronously updating the corresponding mapping relationship in the second array; shrinking the working interval and recording reversible information includes: reducing the range of the working interval by one, and pushing the target index onto the rollback stack.

[0054] Specifically, when a bucket needs to be removed, the removal state is updated in the following three steps through the state compression layer, and the time complexity of the whole process is O(1): Location: Given the bucket number (pos) of the bucket to be removed, query the index position of this bucket number in the first array (bucket_at array) using the second array (pos_of array), i.e., set the target index (idx) = pos_of [pos]. This can quickly pinpoint the exact location of the bucket to be removed within the first array (bucket_at array).

[0055] Replacement with the last slot of the work interval: First, obtain the last index of the current work interval. The last index of the work interval (last) = the total number of currently valid work nodes (W) - 1, where the total number of currently valid work nodes (W) is the number of current work buckets. Swap the value at the target index (idx) in the first array (bucket_at array) with the value at the last index of the work interval (last). Simultaneously, update the index position of the corresponding bucket number in the second array (pos_of array). That is, if the bucket number of the first array (bucket_at array) [last index of the work interval (last)] is b, then update the value of [b] in the second array (pos_of array) to the target index (idx). This ensures that the inverse mapping relationship between the two arrays always holds and prevents logical confusion.

[0056] Shrink the working range and record reversible information: Decrease the current working range by one, i.e., the total number of currently valid working nodes (W) = the total number of currently valid working nodes (W) - 1, completing the dynamic adjustment of the working range. At the same time, push the located target index (idx) onto the rollback stack (rb_stack), rollback stack (rb_stack).push(target index (idx)), completely recording the array indices affected in this removal operation, providing a reversible basis for subsequent bucket recovery operations.

[0057] To more clearly demonstrate the advantages of the state compression layer of this invention, a comparison is made here with traditional methods for removing state records: Traditional methods typically use a replacement set (R) to record the state of the removed buckets; for example, the MementoHash algorithm uses... The tuple records a single replacement operation, where b is the bucket number removed, c is the bucket number replaced, and p is the bucket number previously removed. Taking the scenario of randomly removing bucket numbers 3→5→6→1→2 as an example, the replacement set (R) in the traditional method is... Its state record structure is as follows Figure 8 As shown. When performing a lookup operation, the first node number of the primary mapping result is... If the first node number If the bucket corresponding to the first node number b0 is a removed node, then it is necessary to traverse the tuples in the replacement set (R) sequentially to find the final working bucket. For example, the first node number... At this time, it is necessary to query step by step along the path 1→3→6→4 to locate working bucket 4. This data addressing process is as follows: Figure 9 As shown, when the proportion of storage nodes removed is high, the traversal path will increase synchronously with the number of removed buckets, resulting in a serious degradation of search efficiency.

[0058] The removal operation of this invention does not rely on the replacement set (R). Even if buckets are removed in a random order of 3→5→6→1→2, the process still strictly follows the three steps of positioning → replacing with the last slot in the work interval → shrinking the work interval. For example, if the initial total number of storage nodes N=7, after removing the first bucket in a last-in-first-out order, the bucket size is reduced by 1, and the current total number of valid work nodes W=the initial total number of storage nodes N=6. At this time, the value range of the bucket number (pos) to be removed is [0,6-1]. The array update state is as follows. Figure 6 As shown; after all removals of 3→5→6→1→2 are completed, the first array (bucket_at array) [target index (idx)] within the range of the current total number of valid worker nodes (W) are all valid worker bucket numbers, with no invalid states or redundant records. The search path is completely compressed, as shown in the specific state. Figure 10 As shown.

[0059] By employing a three-step process of locating, replacing, and shrinking, efficient recording and updating of bucket removal states are achieved, maintaining a time complexity of O(1) regardless of the size of the storage nodes or the removal ratio. Through direct swapping of inverse arrays and index recording of the rollback stack, state structure bloat is avoided, and memory usage is linearly related only to the total number of initial storage nodes. Combined with the scenario comparison in the attached diagram, the core advantages of this embodiment—compact search path and no additional traversal overhead—are clearly demonstrated in random node removal scenarios.

[0060] Furthermore, based on the above embodiments, this embodiment further includes the following steps before performing the process of shrinking the working interval and recording reversible information: determining that the target index is equal to the end index of the working interval; marking the value at the target index in the first array as removed; and performing the process of shrinking the working interval and recording reversible information.

[0061] Specifically, determine whether the target index (idx) obtained from the location is equal to the last index (last) of the current working interval. The last index of the working interval is W-1. If idx = last, meaning the bucket to be removed is the last bucket of the current working interval, then the "replace with the last slot of the working interval" process is unnecessary. Instead, mark the value at the target index (idx) in the first array (bucket_at array) as removed (e.g., assign a value of -1), and then execute the "shrink working interval and record reversible information" step. If idx ≠ last, then perform the subsequent operations according to the "replace with the last slot of the working interval" process.

[0062] The process for removing tail buckets is optimized to reduce unnecessary array swapping operations and further improve the efficiency of the removal operation. At the same time, clear marking ensures that the status of removed buckets can be accurately identified, avoiding misjudgments during addressing.

[0063] Furthermore, based on the above embodiments, this embodiment also includes: popping the index recorded by the most recent removal operation from the top of the rollback stack; and restoring the storage node corresponding to the index recorded by the most recent removal operation to the working state and expanding the working range by performing steps that are the opposite of the removal operation logic, according to the index recorded by the most recent removal operation.

[0064] Specifically, when a bucket needs to be added and a bucket has already been removed, the index position recorded in the most recent removal operation is popped from the top of the rollback stack (rb_stack), denoted as the recovery index recover_idx. The recovery index position is the index of the first array (bucket_at array) affected during removal. Based on the recovery index recover_idx, the logic is executed in complete reverse to the removal operation in the above embodiment. If array element swapping was performed during removal, the element at the corresponding position in the first array (bucket_at array) is swapped back to its original state during recovery. If only the removed state was marked during removal, the element at that position is restored to the corresponding bucket number during recovery. The index position of the corresponding bucket number in the second array (pos_of array) is updated synchronously to ensure that the mapping relationship between the two inverse arrays is restored correctly. The range of the current working interval is incremented by one (i.e., W=W+1), completing the recovery of the bucket and allowing it to re-enter the working state.

[0065] The recovery operation relies entirely on the information recorded in the rollback stack (rb_stack), without needing to recalculate the mapping relationship, and the time complexity remains O(1). Figure 7 As can be seen from the newly added storage nodes, restoring the buckets according to the last-in-first-out order of the rollback stack (rb_stack) (e.g., 2→1→6→5→3) can accurately restore the previous working state and ensure the continuity of data mapping.

[0066] Furthermore, based on the above embodiments, this embodiment also includes: determining that the number of current working nodes is equal to the total number of storage nodes, adding the newly added storage nodes to the end of the storage node list, and incrementing the count of the total number of storage nodes.

[0067] Specifically, before performing the bucket addition operation, it is first determined whether the current number of worker nodes (W) is equal to the total number of storage nodes and the initial total number of storage nodes (N).

[0068] If the current total number of active working nodes (W) equals the initial total number of storage nodes (N), meaning all buckets are in working condition and no removed buckets can be recovered, then the newly added bucket is directly added to the end of the storage node list. At this time, the total number of storage nodes increases by 1 from the initial total number of storage nodes (N), i.e., N = N + 1.

[0069] Synchronously update the first array (bucket_at array) and the second array (pos_of array). Add an index position at the end of the first array (bucket_at array) and assign it the number of the new bucket, i.e., the first array (bucket_at array) [total number of initial storage nodes (N) - 1] = the total number of initial storage nodes (N) - 1. Add an index position at the end of the second array (pos_of array) and assign it the index of the new bucket in the first array (bucket_at array), i.e., the second array (pos_of array) [total number of initial storage nodes (N) - 1] = the total number of initial storage nodes (N) - 1. Keep the rollback stack (rb_stack) state unchanged, and the new bucket directly enters the working state to participate in subsequent addressing mapping.

[0070] By covering new scenarios where there are no unrecoverable buckets, the number of buckets can be flexibly expanded. At the same time, new operations only require appending elements to the end of the array, without affecting existing mapping relationships, thus ensuring the scalability and stability of the system.

[0071] This invention employs stateless consistent hashing as the primary mapping and constructs a state compression layer to compress and optimize the search path after node removal. It achieves the high search performance of stateless consistent hashing while supporting the removal of random storage buckets with low memory consumption. By introducing a state compression layer, two inverse arrays and a rollback stack are used to compactly encode the bucket removal state. This supports arbitrary storage removal without altering the underlying algorithm's fundamental mapping behavior, fully leveraging the algorithm's excellent query performance while also supporting the removal of random storage nodes.

[0072] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0073] like Figure 11As shown, the electronic device may include: a processor 1110, a communications interface 1120, a memory 1130, and a communications bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other through the communications bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a state-compressed consistent hashing method that supports arbitrary removal of evidence storage nodes. This method includes: constructing a state compression layer to record the removal status of storage nodes, the state compression layer including a reciprocal array and a rollback stack; when a storage node is removed, updating the removal status through the state compression layer in constant time complexity; addressing data keys based on the removal status recorded by the state compression layer, wherein a primary mapping is performed on the data keys based on a stateless consistent hashing engine to obtain a first node number; when the storage node corresponding to the first node number is in a working state, outputting the first node number as the addressing result; or, when the storage node corresponding to the first node number is in a removed state, triggering a secondary mapping, recalculating based on the first node number in the currently working storage node set, and outputting a second node number as the addressing result, where the storage node corresponding to the second node number is in a working state.

[0074] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the state compression consistent hashing method for supporting arbitrary removal of evidence storage nodes provided by the above methods. The method includes: constructing a state compression layer for recording the removal state of storage nodes, the state compression layer including a reciprocal array and a rollback stack; when a storage node is removed, updating the removal state through the state compression layer with constant time complexity; addressing the data key based on the removal state recorded by the state compression layer, wherein, based on a stateless consistent hashing engine, a primary mapping is performed on the data key to obtain a first node number; when the storage node corresponding to the first node number is in a working state, the first node number is output as the addressing result; or, when the storage node corresponding to the first node number is in a removed state, a secondary mapping is triggered, and a second node number is output as the addressing result based on the first node number in the currently working storage node set, the storage node corresponding to the second node number is in a working state.

[0076] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a state-compressed consistent hashing method for supporting arbitrary removal of evidence storage nodes provided by the methods described above. This method includes: constructing a state compression layer for recording the removal state of storage nodes, the state compression layer including a reciprocal array and a rollback stack; when a storage node is removed, updating the removal state through the state compression layer in constant time complexity; addressing a data key based on the removal state recorded by the state compression layer, wherein a primary mapping is performed on the data key based on a stateless consistent hashing engine to obtain a first node number; when the storage node corresponding to the first node number is in a working state, outputting the first node number as the addressing result; or, when the storage node corresponding to the first node number is in a removed state, triggering a secondary mapping, recalculating based on the first node number in the currently working storage node set, and outputting a second node number as the addressing result, where the storage node corresponding to the second node number is in a working state.

[0077] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A state-compressed consistent hashing method that supports arbitrary removal of evidence storage nodes, characterized in that, include: A state compression layer is constructed to record the removal status of storage nodes. The state compression layer includes a reciprocal array and a rollback stack. When a storage node is removed, the removal status is updated in constant time complexity through the state compression layer. Based on the removal status recorded by the state compression layer, the data key is addressed. A stateless consistent hash engine is used to perform a primary mapping on the data key to obtain a first node number. When the storage node corresponding to the first node number is in a working state, the first node number is output as the addressing result. Alternatively, when the storage node corresponding to the first node number is in a removed state, a secondary mapping is triggered. Based on the first node number, a second node number is recalculated in the currently working storage node set, and the second node number is output as the addressing result. The storage node corresponding to the second node number is in a working state.

2. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 1, characterized in that, After outputting the first node number as the addressing result, the method further includes: If the state corresponding to the node with the first node number stored in the state compression layer is determined to be a working state, the search is terminated and the first node number is output; or... If it is determined that the state corresponding to the node with the first node number stored in the state compression layer is a removed state, then a secondary mapping is triggered.

3. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 2, characterized in that, The step of recalculating the second node number based on the first node number in the current working node set and outputting the second node number as the addressing result includes: The first node number is used as input, and the intermediate hash value is calculated using a preset hash function; The intermediate hash value is moduloed by the total number of currently valid working nodes to obtain the second node number as the addressing result.

4. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 1, characterized in that, The constructed state compression layer includes: Create a first array to store the numbers of the storage nodes in index order; Create a second array that is the inverse of the first array, and use it to store the index position of the number of each storage node in the first array.

5. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 4, characterized in that, Also includes: Based on the total number N of initial storage nodes, assign a storage node number value to each index position i of the first array, such that the first array [i] = i, where i ∈ [0, N); Assign an index position value to each storage node number i of the second array, such that the second array [i] = i, and establish an initial inverse mapping relationship between node numbers and array indices.

6. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 5, characterized in that, Also includes: A rollback stack is provided to record array indices of content replacements caused by storage node removal operations. When a node needs to be restored, the array indices are popped from the rollback stack to guide the storage node to add the reverse operation.

7. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 6, characterized in that, Removing a storage node while updating the removal status includes: locating it, replacing it with the last slot in the work area, shrinking the work area and recording reversible information; The location includes: querying the second array based on the node number to be removed, and obtaining the target index of the node number to be removed in the first array; The replacement with the end slot of the work interval includes: modifying the value of the node number to be removed in the first array to the end index of the work interval, swapping the value at the target index in the first array with the value at the end index of the work interval, and synchronously updating the corresponding mapping relationship in the second array; The shrinking of the working interval and recording of reversible information includes: reducing the range of the working interval by one and pushing the target index onto the rollback stack.

8. The state compression consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 7, characterized in that, Before performing the process of shrinking the working interval and recording reversible information, the following steps are also included: The process involves determining that the target index is equal to the end index of the working interval, marking the value at the target index in the first array as removed, and then performing the shrinking of the working interval and recording reversible information.

9. The state-compressed consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 6, characterized in that, Also includes: Pop the index recorded for the most recent removal operation from the top of the rollback stack; Based on the index recorded in the most recent removal operation, by performing steps that are the reverse of the removal operation logic, the storage node corresponding to the index recorded in the most recent removal operation is restored to the working state, and the working range is expanded.

10. The state-compressed consistent hashing method supporting arbitrary removal of evidence storage nodes according to claim 9, characterized in that, Also includes: Determine that the current number of working nodes equals the total number of storage nodes, add the newly added storage node to the end of the storage node list, and increment the total number of storage nodes count.

Images