Storage metadata synchronization method and device, electronic equipment and storage medium

Abstract

The invention discloses a synchronization method and device for storage metadata, electronic equipment and a storage medium, and relates to the technical field of computers, a consistent Hash algorithm is adopted to realize mapping of data fragments and storage nodes, a monitoring node maintains the relation, a data change event is directionally sent to a corresponding partition, and the data change event is stored in the corresponding partition. Meanwhile, the Bloom filter is used for pre-checking events to filter invalid information, only local related changes are processed, the check value is transmitted, and the memory area is pre-registered, so that the data interaction mode between the nodes is optimized, and the defects of a full-connection mode are avoided; the technical problems that in an existing block storage metadata synchronization mechanism, due to the fact that a full connection mode is not optimized for node scale extension, the network connection number is increased in an exponential level mode, the system overhead is increased, and the cluster expansion capacity is limited can be solved, and the purposes of reducing the system overhead, improving the cluster expansion capacity and improving the cluster expansion efficiency are achieved. And the performance of the distributed block storage system is optimized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, and storage medium for synchronizing stored metadata. Background Art

[0002] With the continuous development of cloud computing and big data applications, distributed block storage systems, as a core component of data center storage architecture, are widely used in enterprise-level storage, virtualization platforms, high-performance computing and other fields.

[0003] Currently, the native architecture of the block storage metadata synchronization mechanism directly adopts a fully connected mode and is not optimized for node scale expansion. This may lead to an exponential growth in the number of network connections, significantly increasing system overhead and limiting cluster scalability. Summary of the Invention

[0004] The present application provides a method, device, electronic device and storage medium for synchronizing stored metadata to at least solve the problem in related technologies that the number of network connections grows exponentially, significantly increases system overhead and limits cluster expansion capabilities.

[0005] This application provides a method for synchronizing stored metadata, including:

[0006] Map data shards to target object storage device nodes based on the consistent hashing algorithm, and maintain the mapping relationship between shards and object storage device nodes through monitoring nodes;

[0007] Encapsulate the data change event as a message body and send it to the partition corresponding to the target shard;

[0008] Pre-check received events against the locally cached Bloom filter, filter out invalid shard events, and only receive change events related to the local shard;

[0009] Transmits the checksum of the changed data block and pre-registers the memory area.

[0010] Optionally, mapping the data shards to the target object storage device nodes based on the consistent hashing algorithm and maintaining the mapping relationship between the shards and the object storage device nodes through the monitoring node further includes:

[0011] Calculate the hash value of the data block based on the preset algorithm, and determine the target shard through a consistent hash ring with the number of virtual nodes equal to the number of physical nodes;

[0012] Dynamically adjust the shard mapping strategy based on the storage type of the object storage device node.

[0013] Optionally, encapsulating the data change event into a message body and sending it to the partition corresponding to the target shard also includes:

[0014] The message body includes a fragment identifier, event type, version number and check value;

[0015] Events are sent to the partitions via an asynchronous publishing mechanism, where each subscriber group corresponds to at least one consumer node.

[0016] Optionally, pre-checking the received events according to the locally cached Bloom filter, filtering out invalid shard events, and then receiving only change events related to the local shards further includes:

[0017] The Bloom filter synchronizes the latest shard list based on a preset time interval.

[0018] Optionally, the method further includes:

[0019] In response to a shard synchronization failure, an exponential backoff algorithm is employed to synchronize the shards.

[0020] Optionally, the method further includes:

[0021] Shard migration is triggered when the load of the object storage device node exceeds the preset threshold, a new node joins the cluster, or an existing node fails and goes offline.

[0022] Optionally, the transmission of the check value of the changed data block adopts an incremental synchronization protocol, and only the check value of the changed data block is transmitted to generate a differential patch.

[0023] The present application also provides a synchronization device for storing metadata, including:

[0024] A mapping unit, configured to map data shards to target object storage device nodes based on a consistent hashing algorithm, and maintain a mapping relationship between shards and object storage device nodes through a monitoring node;

[0025] The sending unit is used to encapsulate the data change event into a message body and send it to the partition corresponding to the target shard;

[0026] A receiving unit, configured to pre-check received events based on a locally cached Bloom filter, filter out invalid shard events, and only receive change events related to the local shard;

[0027] The transmission unit is used to transmit the check value of the changed data block and pre-register the memory area.

[0028] Optionally, the mapping unit is further configured to:

[0029] Calculate the hash value of the data block based on the preset algorithm, and determine the target shard through a consistent hash ring with the number of virtual nodes equal to the number of physical nodes;

[0030] Dynamically adjust the shard mapping strategy based on the storage type of the object storage device node.

[0031] Optionally, the sending unit is further used to:

[0032] The message body includes a fragment identifier, event type, version number and check value;

[0033] Events are sent to the partitions via an asynchronous publishing mechanism, where each subscriber group corresponds to at least one consumer node.

[0034] Optionally, the receiving unit is further configured to:

[0035] The Bloom filter synchronizes the latest shard list based on a preset time interval.

[0036] Optionally, the device further includes:

[0037] The sharding unit is configured to perform shard synchronization using an exponential backoff algorithm in response to a shard synchronization failure.

[0038] Optionally, the device further includes:

[0039] The migration unit is used to trigger shard migration when the load of the object storage device node exceeds the preset threshold, a new node joins the cluster, or an existing node fails and goes offline.

[0040] Optionally, the transmission of the check value of the changed data block adopts an incremental synchronization protocol, and only the check value of the changed data block is transmitted to generate a differential patch.

[0041] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for synchronizing stored metadata when executing the computer program.

[0042] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for synchronizing stored metadata are implemented.

[0043] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned methods for synchronizing stored metadata when executed by a processor.

[0044] Through this application, since a consistent hashing algorithm is used to realize the mapping between data shards and storage nodes and the relationship is maintained by the monitoring node, data change events are directed to the corresponding partitions, and a Bloom filter is used to pre-check events to filter out invalid information. Only local related changes are processed and check values ​​and pre-registered memory areas are transmitted. This optimizes the data interaction mode between nodes and avoids the disadvantages of the full connection mode. Therefore, it can solve the technical problem in the existing block storage metadata synchronization mechanism that the full connection mode is not optimized for node scale expansion, resulting in an exponential growth in the number of network connections, increased system overhead and limited cluster expansion capabilities, thereby achieving the technical effect of reducing system overhead, improving cluster expansion capabilities and optimizing the performance of distributed block storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 A flowchart of a method for synchronizing stored metadata provided by an embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram of a technology integration framework provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a data shard mapping process provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a data transmission process provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of an event publishing mechanism provided in an embodiment of the present application;

[0051] Figure 6 A schematic structural diagram of a synchronization device for storing metadata provided by an embodiment of the present disclosure;

[0052] Figure 7 A schematic structural diagram of another synchronization device for storing metadata provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0055] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0056] An embodiment of the present application provides a method for synchronizing stored metadata. The method is described in detail in conjunction with the execution flow of the method for synchronizing stored metadata. Figure 1 A flowchart of a method for synchronizing stored metadata provided by an embodiment of the present disclosure is provided.

[0057] like Figure 1 As shown, the method comprises the following steps:

[0058] Step 101: Map data shards to target object storage device nodes based on a consistent hashing algorithm, and maintain the mapping relationship between shards and object storage device nodes through a monitoring node;

[0059] To achieve uniform distribution and dynamic adjustment of data shards across OSD nodes, the system constructs a consistent hash ring. This hash ring is built using the MurmurHash3 (mmh3) algorithm, a high-performance hash algorithm widely used in distributed systems. Its low collision rate and uniform distribution make it ideal for generating shard keys. In practice, the mmh3.hash64 method calculates a 64-bit hash value (hash_val) for the input data block identifier (block_id), which serves as the basis for data shard mapping.

[0060] To ensure uniform distribution of shards across physical OSD nodes, the system introduces a virtual node mechanism. The number of virtual nodes is set to 16 times the number of physical OSD nodes. By assigning multiple virtual nodes to one physical node, this effectively avoids the issue of large-scale shard migrations caused by the online and offline status of individual physical nodes. After obtaining a 64-bit hash value (hash_val), the system performs a modulo operation on the hash value (% len(vnodes), where vnodes is the list of virtual nodes) to obtain an integer index (vnode_idx) in the range [0, len(vnodes)-1]. Based on this index, the system retrieves the corresponding virtual node identifier from the virtual node list. This virtual node identifier is associated with a specific physical OSD node (e.g., represented by an IP:port or resource identifier), ultimately determining the target OSD node to which the data shard belongs. Furthermore, the default sharding granularity is 64MB, which can be dynamically adjusted based on actual business needs to accommodate data storage and access requirements in different scenarios.

[0061] At the same time, the Monitor node is responsible for maintaining the mapping relationship between shards and OSD nodes. When the OSD node starts, it will actively register its own hardware capability information, including the storage media type (such as SSD, HDD, NVMe, etc.) with the Monitor node, so that the Monitor node can allocate shards according to the actual capabilities of each OSD node. The Monitor node generates an initial shard mapping table based on the CRUSH algorithm. The mapping table records the association between each data shard and the corresponding OSD node, and stores it in its own Key-Value database to achieve persistent storage of the mapping relationship. During system operation, if there is a change in OSD node load, a new node is added, or an existing node fails and goes offline, the Monitor node will dynamically update the shard mapping table according to the actual situation, and synchronize it to each OSD node through the corresponding mechanism to ensure that the mapping relationship between shards and OSD nodes in the entire system is always accurate and effective.

[0062] Step 102: Encapsulate the data change event into a message body and send it to the partition corresponding to the target shard;

[0063] When the OSD node completes the data persistence operation, it generates a corresponding data change event (such as SyncEvent or BlockEvent). This event contains key information such as the data shard identifier (shard_id), the change content, and the version number, which is used to accurately describe the data change. To achieve efficient transmission and parsing of events, the system uses the Protobuf serialization method to process data change events. As a lightweight and efficient structured data storage format, Protobuf can convert event objects into compact binary byte arrays, which not only reduces the amount of data transmission, but also ensures cross-platform and cross-language compatibility, ensuring that different nodes can correctly parse the event content.

[0064] After completing the serialization of the event, the system will construct a Kafka production record (ProducerRecord) as the message body. The message body clearly contains three core elements: the first is the topic (Topic), which is uniformly designated as "block_events" and is used to centrally manage the change events of all data shards, so that subscribers can monitor them uniformly; the second is the key (Key), which is set to the shard identifier (event.getShardId()) in the event. This setting ensures that all change events belonging to the same data shard are written to the corresponding partition in sequence, avoiding the disorder of event order caused by chaotic partition allocation; the third is the value (Value), which is the binary byte array of the event after Protobuf serialization, which serves as the core data content of the message body.

[0065] Subsequently, the system asynchronously sends the message body by calling the send() method of the Kafka producer. Under the asynchronous sending mechanism, the Kafka producer will temporarily store the message body in a local buffer and then send it in batches to the KafkaBroker through a background thread. This method can reduce the blocking of the sending operation on the front-end business and improve the system's concurrent processing capabilities. At the same time, to ensure that the message body is sent to the partition corresponding to the target shard, the system adopts the "hash(shard_id) mod partition_count" partition allocation strategy, where hash(shard_id) is the result of the hash operation on the shard identifier, and partition_count is the total number of partitions under the topic. Through the modulo operation, events of the same shard can be stably mapped to fixed partitions, ensuring the accuracy and consistency of event distribution, so that nodes subscribing to the shard can accurately obtain the corresponding change events.

[0066] Step 103: Pre-check the received events based on the locally cached Bloom filter, filter out invalid shard events, and only receive change events related to the local shard;

[0067] The system maintains a Bloom filter instance locally on each subscription node. This filter serves as the core tool for event pre-checking, and is used to quickly determine whether the received event may be related to the locally managed shards, thereby efficiently filtering out irrelevant and invalid events.

[0068] Bloom filter initialization requires setting key parameters: the first is capacity (capacity), which defaults to 10,000 to ensure that it can cover the maximum shard management capacity of the current node and meet the storage needs of the daily number of shards; the second is the error rate (error_rate), which is set to 0.001, that is, the probability of "false positive" does not exceed 0.1%. This setting ensures filtering efficiency while keeping the probability of invalid events misjudged as valid events to an extremely low level, reducing subsequent unnecessary processing overhead. After initialization is complete, the system will traverse all locally managed shard identification lists (shard_list), map each shard ID (sid) to the Bloom filter's bit array through multiple hash functions, mark these bits as "present", complete the pre-loading of shard IDs, and enable the filter to accurately identify events related to local shards.

[0069] When a node receives an event, it first extracts the shard identifier (shard_id) from the event and inputs it into the locally cached Bloom filter for detection. If the Bloom filter determines that the shard identifier does not exist, the event is directly determined to be an invalid shard event and filtered out without further processing; if it is determined to exist (allowing an extremely low probability of false positives), the event is considered to be possibly related to the local shard and is retained for further verification. Through this pre-check mechanism, a large number of invalid events unrelated to the local shard are filtered out in advance, significantly reducing the node's processing overhead for invalid events and improving event processing efficiency.

[0070] To ensure that the shard information recorded in the Bloom filter is consistent with the shards actually managed by the node, the system sets up a filter update mechanism to synchronize the latest local shard list every 10 seconds, update the Bloom filter, add the newly added shard ID to the filter, and remove the mark corresponding to the shard ID that is no longer managed locally, to ensure that the filter can always accurately reflect the shard management status of the current node, further improving the accuracy of invalid event filtering.

[0071] Step 104: Transmit the checksum of the changed data block and pre-register the memory area.

[0072] During data synchronization, to reduce network transmission volume and ensure data accuracy, the system uses an incremental synchronization protocol, transmitting only the XOR checksum of the changed data blocks. Specifically, for the source data block (src_data) and the target data block (dst_data), the system generates a differential patch through a byte-by-byte XOR operation: first, a bytearray object (xor_result) of the same length as the source or target data block is created as patch storage space; then, each byte index i (from 0 to the data block length - 1) is traversed, performing an XOR operation (^) on src_data[i] and dst_data[i], and storing the result in xor_result[i]. Due to the characteristics of the XOR operation, each bit position in the result that is 1 indicates a difference between the source and target data at that binary bit, while a 0 indicates an identical bit. Therefore, xor_result accurately represents the binary difference between the two data blocks, namely the checksum of the changed data block. Its average size is less than 64KB, significantly reducing network transmission overhead.

[0073] To improve data transmission efficiency, the system uses RDMA (Remote Direct Memory Access) zero-copy technology and reduces RDMA operation latency by pre-registering memory regions. Pre-registering memory regions means that before RDMA communication begins, the local host registers the memory regions to be transferred (especially hotspot data regions) with the RDMA network adapter (HCA) and grants access rights to the remote host. This process avoids frequent memory registration and permission verification during data transmission, reduces initialization overhead, and, after optimization, reduces RDMA initialization latency. When data transmission occurs, the local host submits a write request using ibv_post_send, containing information such as the data address and length of the pre-registered memory region, as well as the remote memory address. The RDMA network adapter removes the request from the send queue and directly transmits the XOR checksum to the remote host's pre-registered memory region via the RoCE v2 protocol (RDMA over IP). This eliminates CPU intervention and kernel-mode and user-mode data copying, achieving efficient zero-copy transmission and ensuring the rapid and accurate transmission of checksums for changed data blocks.

[0074] In some embodiments, mapping the data shards to the target object storage device nodes based on the consistent hashing algorithm and maintaining the mapping relationship between the shards and the object storage device nodes through the monitoring node further includes:

[0075] Calculate the hash value of the data block based on the preset algorithm, and determine the target shard through a consistent hash ring with the number of virtual nodes equal to the number of physical nodes;

[0076] Dynamically adjust the shard mapping strategy based on the storage type of the object storage device node.

[0077] When mapping data shards to target object storage device (OSD) nodes based on the consistent hashing algorithm and maintaining the mapping relationship between shards and OSD nodes through the monitor node, the hash value of the data block is calculated based on the preset algorithm, and the target shard is determined through a consistent hash ring with a virtual node number that is a multiple of the physical node number. At the same time, the shard mapping strategy is dynamically adjusted according to the storage type of the OSD node. The specific implementation is as follows:

[0078] In terms of calculating the hash value of the data block, the system uses the preset MurmurHash3 (mmh3) algorithm, which is a high-performance hash algorithm widely used in distributed systems. It has the characteristics of low collision rate and uniform distribution, and can provide a stable and reliable mapping basis for data sharding. Specifically, for the input data block identifier (block_id), a 64-bit hash value (hash_val) is calculated and generated by the mmh3.hash64 method. This hash value serves as the basis for locating the data shard on the consistent hash ring. In order to ensure that the data shards are evenly distributed on the physical OSD nodes, the consistent hash ring constructed by the system introduces a virtual node mechanism. The number of virtual nodes is set to 16 times the number of physical nodes. By using multiple virtual nodes to correspond to one physical node, the problem of large-scale migration of shards caused by the online and offline physical nodes is effectively reduced. After obtaining the 64-bit hash value, the system performs a modulo operation on it (% len(vnodes), where vnodes is the virtual node list) to obtain the virtual node index (vnode_idx), and then obtains the corresponding virtual node identifier from the virtual node list based on the index, and then determines the target OSD node to which the data block belongs, thereby achieving accurate mapping of data shards to target nodes.

[0079] In terms of dynamically adjusting the shard mapping strategy based on the storage type of the OSD node, when the OSD node starts, it will actively register its own storage type information with the Monitor node, including SSD (solid-state drive), HDD (mechanical hard disk), NVMe (non-volatile memory host controller interface specification), etc. This information is stored by the Monitor node in its Key-Value database as an important basis for shard mapping adjustment. When the Monitor node generates or updates the shard mapping table based on the CRUSH algorithm, it will dynamically allocate it based on the storage type characteristics of each OSD node: for hot data shards with high access frequency and high read and write speed requirements, they will be mapped to SSD or NVMe nodes with high-speed read and write capabilities to improve data access efficiency; for cold data shards with low access frequency and high storage capacity requirements, they will be allocated to HDD nodes with large storage capacity and low cost to optimize storage resource utilization. At the same time, when the system detects a change in the frequency of shard access (such as hot data becoming cold or cold data becoming hot), the Monitor node will trigger the shard migration mechanism based on the latest storage type information and shard access characteristics, and adjust the mapping relationship between the shard and the OSD node to ensure that the shard is always associated with the storage type node that best suits its characteristics, thereby improving system performance while achieving reasonable configuration of storage resources.

[0080] In some embodiments, encapsulating the data change event into a message body and sending it to the partition corresponding to the target shard further includes:

[0081] The message body includes a fragment identifier, event type, version number and check value;

[0082] Events are sent to the partitions via an asynchronous publishing mechanism, where each subscriber group corresponds to at least one consumer node.

[0083] When a data change event is encapsulated as a message body and sent to the partition corresponding to the target shard, the message body contains the shard identifier, event type, version number, and checksum. It is sent through an asynchronous publishing mechanism. Each subscriber group corresponds to at least one consumer node. The specific implementation is as follows:

[0084] The message body is encapsulated around a binary byte array serialized with Protobuf, which clearly contains four key pieces of information: the shard ID (shard_id), which uniquely associates the event with the target shard, ensuring that the event can be accurately located on the corresponding shard; the event type, which distinguishes between different types of operations, such as data writes, deletes, and updates, allowing the recipient to clearly understand the event processing logic; the version number, which uses a monotonically increasing mechanism (e.g., the version number increases by 1 after each metadata change) to ensure the directionality and consistency of synchronization operations and avoid synchronization errors caused by version conflicts; and the checksum, generated using a hash algorithm such as CRC32, which quickly detects data corruption or tampering during transmission due to packet loss, interference, and other factors, ensuring the integrity of the event data. After being serialized with Protobuf, this information constitutes the "Value" portion of the message body, which, together with the "Key" (i.e., shard ID) and "Topic" (i.e., block_events), forms the Kafka ProducerRecord, forming the complete message body.

[0085] The system uses an asynchronous publishing mechanism: calling the send() method on a Kafka producer does not block the current business process, but instead temporarily stores the message body in a local buffer. This buffer triggers batch sending based on preset thresholds (such as message count or size), and a background thread sends all the message bodies in the buffer to the Kafka broker. This approach reduces the overhead of frequent network interactions, improves event publishing efficiency, and avoids blocking front-end data processing.

[0086] To ensure that events can be consumed effectively, each subscriber group (such as a cluster of replica nodes across data centers) corresponds to at least one consumer node. The consumer node is responsible for monitoring the partitions associated with the subscriber group, receiving and processing events therein. Even in the minimum configuration, a single consumer node can take on the task of consuming events for the subscriber group. When the number of events increases, the load can be shared by increasing the number of consumer nodes (up to 1,000 is supported), ensuring the timeliness of event processing. This design not only ensures the availability of the subscription link, but also provides flexibility for system expansion, allowing events to be stably received and processed by the subscriber group.

[0087] In some embodiments, pre-checking received events based on the locally cached Bloom filter, filtering invalid shard events, and then receiving only change events related to the local shard further includes:

[0088] The Bloom filter synchronizes the latest shard list based on a preset time interval.

[0089] According to the Bloom filter pre-checking the received events based on the local cache, when only the change events related to the local shards are received after filtering out the invalid shard events, the Bloom filter synchronizes the latest shard list based on the preset time interval. The specific implementation process is as follows: In the distributed storage system, shards may migrate due to load balancing, node failure, new node joining, etc., resulting in dynamic changes in the shard list (shard_list) managed locally by the node. In order to ensure that the Bloom filter can accurately identify events related to the current local shard, the system sets a fixed synchronization period with a preset time interval of every 10 seconds, that is, the Bloom filter is updated every 10 seconds to keep it consistent with the latest shard list.

[0090] During each synchronization cycle, the node will first obtain the latest local shard list, which contains all the shard identifiers (shard_id) currently managed by this node. Subsequently, the system starts the Bloom filter update process: on the one hand, it traverses the newly added shard identifiers and maps these new shard IDs to the bit array through the multiple hash functions of the Bloom filter, marking the corresponding bits as "existing" to ensure that the events of the new shards can be recognized by the filter; on the other hand, for shard identifiers that have been removed from the local shard list (such as shards that have been migrated to other nodes), although the Bloom filter cannot directly delete their marks (due to the limitations of the Bloom filter characteristics), by reloading the latest shard list and updating the valid mark range of the bit array, it can indirectly weaken the impact of the old shard identifiers and reduce misjudgments caused by residual old shard information.

[0091] This synchronization mechanism, based on preset time intervals, ensures that the Bloom filter promptly reflects dynamic changes in shards, preventing valid events from being incorrectly filtered or invalid events from being missed due to shard migrations. It also controls the synchronization frequency through a fixed period, avoiding the performance overhead associated with high-frequency synchronization. By regularly updating every 10 seconds, the Bloom filter consistently maintains accurate identification of the current local shard, further improving the accuracy of invalid event filtering and ensuring that nodes only receive and process change events related to the shards they manage, effectively reducing system processing pressure.

[0092] In some embodiments, the method further comprises:

[0093] In response to a shard synchronization failure, an exponential backoff algorithm is employed to synchronize the shards.

[0094] In response to shard synchronization failure, the specific implementation process of shard synchronization using the exponential backoff algorithm is as follows: When a shard synchronization operation (such as changing the check value transmission of the data block, event confirmation feedback, etc.) fails (for example, network interruption, temporary unavailability of the target node, etc.), the system will automatically trigger the retry mechanism. At this time, the exponential backoff algorithm is used to calculate the waiting delay time for each retry to avoid network congestion or target node pressure overload caused by frequent retries in a short period of time, while improving the retry success rate.

[0095] In actual implementation, when shard synchronization fails, the system will first record the current number of retries (attempt), substitute it into the above formula to calculate the delay time for this retry, and then enter a waiting state. After the waiting period is over, the system will re-initiate the shard synchronization operation; if it fails again, the number of retries will be incremented (attempt+1), and the delay time will be recalculated for the next wait and retry, until the synchronization is successful or the preset maximum number of retries is reached (to avoid infinite retries). Through this mechanism, synchronization can be restored through retries in the event of temporary failures such as network fluctuations, and the exponentially increasing delays and random jitter can be used to effectively reduce the system resources and network bandwidth occupied by retry operations, thereby improving the robustness and stability of distributed storage systems in the face of synchronization failures.

[0096] In some embodiments, the method further comprises:

[0097] Shard migration is triggered when the load of the object storage device node exceeds the preset threshold, a new node joins the cluster, or an existing node fails and goes offline.

[0098] When the load of an Object Storage Device (OSD) node exceeds a preset threshold, a new node joins the cluster, or an existing node fails and goes offline, the specific implementation process of triggering shard migration is as follows:

[0099] The system monitors the operating status of the OSD node in real time, and automatically starts the shard migration mechanism when any of the following trigger conditions is met: First, the OSD node load exceeds the preset threshold (the default setting is 85%). At this time, the number of shards carried by the node or the data processing pressure is close to its performance limit. Continuing to maintain the current shard distribution will lead to increased response delays; second, a new node joins the cluster. In order to fully utilize the storage and computing resources of the new node, some existing shards need to be migrated to the new node to achieve load balancing; third, an existing node fails and goes offline. In order to ensure data availability and service continuity, the shards on the failed node need to be migrated to other healthy nodes.

[0100] The shard migration process is mainly divided into three stages: the first is source-side preparation. When the migration is triggered, the source OSD node will first freeze the write permissions of the shard to be migrated to prevent data inconsistency between the source and target ends due to data writing during the migration process; at the same time, a metadata snapshot of the shard is generated, including key content such as the shard's index information, version number, data block mapping relationship, etc., as the baseline data for the migration.

[0101] The second is data transmission. The source OSD node sends a shard migration permission request to the monitor node. The monitor node verifies the legitimacy of the request through the Quorum mechanism and returns a migration token. The source OSD node starts data transmission with the token. The transmission process uses RDMA (remote direct memory access) zero-copy technology to transfer the shard data and metadata snapshot directly from the pre-registered memory area of ​​the source OSD to the corresponding memory area of ​​the target OSD, reducing data copy overhead and improving transmission efficiency. After the target OSD node completes the reception, it returns confirmation information to the source OSD and monitor nodes.

[0102] Finally, the metadata is updated. After receiving confirmation of the transfer completion, the Monitor node updates the CRUSH mapping rules and records the mapping relationship between the migrated shards and the target OSD nodes in the Key-Value database. At the same time, the Quorum mechanism synchronizes metadata updates within the cluster, ensuring that all nodes obtain the latest shard mapping information and completing the entire shard migration process. This dynamic migration mechanism ensures that the system can always maintain load balancing among OSD nodes, improving the overall availability and resource utilization of the cluster.

[0103] In some embodiments, the transmission of the check value of the changed data block adopts an incremental synchronization protocol, and only the check value of the changed data block is transmitted to generate a differential patch.

[0104] The incremental synchronization protocol is used to transmit the checksum of the changed data block. The specific implementation process for generating the differential patch is as follows: This incremental synchronization protocol accurately extracts the differences between the source and target data blocks by leveraging the characteristics of the XOR operation. This allows only the necessary checksum information to be transmitted to generate the differential patch, significantly reducing the amount of data transmitted over the network. Specifically, when a data block change is detected, the system first compares and analyzes the source data block (src_data) and the target data block (dst_data). The first step is to initialize the patch storage space and create a bytearray object (xor_result) of the same length as the source or target data block to store the final XOR result, i.e., the differential patch. The second step is to perform a byte-by-byte XOR calculation, traversing each byte index i of the two data blocks (from 0 to the data block length - 1), performing an XOR operation (^) on src_data[i] and dst_data[i], and storing the result in xor_result[i]. Due to the characteristics of the XOR operation, each bit in the result that is 1 indicates that the source data and the target data are different in that binary bit, and 0 indicates the same. Therefore, xor_result can completely and accurately represent the binary difference between the two blocks of data, that is, the checksum of the changed data block. The average size of the final differential patch is less than 64KB, and it only contains part of the information that has actually changed in the data block, rather than the complete data block content. By transmitting this differential patch, the receiving end can use the checksum to perform a reverse XOR operation with the local data block to restore the complete changed data block and achieve incremental data synchronization. This method effectively reduces network bandwidth usage by 83% and significantly improves the efficiency of cross-node data synchronization. It is particularly suitable for scenarios with frequent data changes in large-scale distributed storage systems.

[0105] To clearly illustrate the method for synchronizing stored metadata provided in an embodiment of the present application, an example is provided below for illustration.

[0106] See also Figure 2 , Figure 2 A schematic diagram of a technology integration framework provided in an embodiment of the present application is shown in FIG. Figure 2 As shown in the figure, this solution builds a distributed block storage synchronization mechanism through the deep integration of the shard registration layer and the subscription distribution layer:

[0107] Shard registration layer: responsible for the dynamic distribution and load balancing of data shards;

[0108] Subscription distribution layer: realizes efficient propagation and filtering of cross-shard events.

[0109] Shard migration mechanism: When the OSD load exceeds the threshold, shard migration is automatically triggered;

[0110] Incremental synchronization protocol: only the XOR checksum of the changed data block is transmitted;

[0111] Subscription filtering algorithm: Reduce invalid event transmission based on Bloom filter.

[0112] See also Figure 3 , Figure 3 A data sharding mapping process diagram provided in an embodiment of the present application is as follows: Figure 3 As shown, the implementation mechanism is:

[0113] Consistent hash ring construction: Use the MurmurHash3 algorithm to calculate the hash value of the data block; the number of virtual nodes = the number of physical nodes × 16 (to ensure uniform distribution of shards); shard granularity: default 64MB (dynamically adjustable).

[0114] Sharding calculation: Hash value calculation: Use the mmh3 (MurmurHash3) hash algorithm to perform a hash operation on the input block_id to generate a 64-bit hash value (hash_val).

[0115] Reason for choosing mmh3.hash64: MurmurHash3 is a high-performance hash algorithm commonly used in distributed systems. It has the characteristics of low collision rate and uniform distribution, making it suitable as the basis for generating shard keys.

[0116] Determine the virtual node index: perform a modulo operation (% len(vnodes)) on the 64-bit hash value (hash_val) to obtain an integer index (vnode_idx) in the range [0, len(vnodes)-1].

[0117] The purpose of the modulo operation is to map consecutive hash values ​​to a limited list of virtual nodes, ensuring that the result falls within the predefined virtual node range.

[0118] Return the target virtual node: According to the calculated index (vnode_idx), obtain the corresponding virtual node identifier (such as the IP: port or resource identifier of the physical node) from the virtual node list (vnodes) as the final sharding of the data block.

[0119] Dynamic registration: When the OSD node starts, it registers its own capabilities (SSD / HDD / NVMe) with the Monitor. The Monitor generates an initial shard mapping table based on the CRUSH algorithm; the shard mapping relationship is stored in the Monitor's Key-Value database.

[0120] Load balancing mechanism: The trigger conditions are: OSD load exceeds the threshold (85% by default); new nodes join the cluster; existing nodes fail and go offline.

[0121] Migration process: Source-side preparation: Freeze shard write permissions; generate shard metadata snapshots.

[0122] See also Figure 4 , Figure 4 A data transmission process diagram provided in an embodiment of the present application is as follows: Figure 4 As shown, it includes: metadata update: updating CRUSH mapping rules through the Monitor node; using the Quorum mechanism to ensure metadata consistency.

[0123] Heartbeat detection mechanism: OSD sends a heartbeat message to Monitor every 3 seconds; the heartbeat message contains the list of currently carried shards and load indicators; the Monitor node determines the health status of the OSD through Quorum voting.

[0124] See also Figure 5 , Figure 5 A schematic diagram of an event publishing mechanism provided in an embodiment of the present application is shown as follows: Figure 5 As shown, it includes: Event serialization: converting the BlockEvent object into a binary byte array through the Protobuf serialization method (Protobuf's default serialization method).

[0125] Construct ProducerRecord: Create a Kafka production record (ProducerRecord), specify the topic (Topic): block_events (all shard events are sent to this topic); key (Key): event.getShardId() (the shard ID is used as the key to ensure that events of the same shard are written to the Kafka partition in sequence); value (Value): the serialized BlockEvent byte array.

[0126] Call the producer.send() method to send events asynchronously. The Kafka producer will temporarily store the events in a local buffer and send them in batches to the Kafka Broker through a background thread.

[0127] Create a Bloom filter instance and set the following parameters: capacity = 10000: This specifies the filter's expected capacity (storing up to 10,000 shard IDs), ensuring that it covers the current node's maximum shard management capacity. error_rate = 0.001: This sets the false positive rate (the probability of a "false positive" does not exceed 0.1%). This means that in extreme cases, a shard may not exist but the filter mistakenly identifies it as existing, but the probability is extremely low (only 1 false positive in 1,000). Preload shard IDs: This loops through each shard ID (sid) in shard_list and adds it to the Bloom filter. The Bloom filter uses multiple hash functions to map the shard IDs to a bit array, marking those bits as "present."

[0128] Version monotonicity: The monotonically increasing version number (for example, the version number increases by 1 each time metadata is changed) ensures the directionality and consistency of synchronization operations, avoiding synchronization errors caused by version number conflicts (such as clock rollback).

[0129] Data integrity verification: CRC32 checksum (or other hash algorithms such as SHA-256) can quickly detect whether data has been damaged during transmission due to packet loss, interference, etc. (for example, a bit flip of a byte), or whether it has been maliciously tampered with.

[0130] Double insurance mechanism: The combined verification of version number and checksum not only avoids the logical error of "old version overwriting new version", but also prevents "dirty data" from polluting local storage, ensuring the consistency and reliability of metadata.

[0131] Topic partition design: Each data shard corresponds to a Kafka topic partition; partition allocation strategy: hash(shard_id) mod partition_count; each subscriber group supports up to 1,000 consumer nodes.

[0132] Incremental synchronization protocol: only transmits the XOR checksum of the changed data block (the average size of each patch is <64KB); supports breakpoint resuming (progress is recorded through the last_sync_version field); subscription filtering algorithm: uses Bloom Filter (128-bit Bloom filter) to filter invalid shard events; filter update frequency: synchronizes the latest shard list every 10 seconds.

[0133] Synchronous process implementation

[0134] Step 1:

[0135] Data writing and shard registration, including: application layer write request: zero copy write to OSD cache; triggering shard registration.

[0136] The shard registration process includes: after the OSD node receives a write request, it calculates the shard (shard_id) to which the data block belongs; updates the consistent hash ring mapping relationship through the Monitor node; and uses the Raft protocol to ensure strong consistency for metadata updates.

[0137] Step 2:

[0138] Event notification and subscription distribution: After OSD completes data persistence, it generates a SyncEvent event.

[0139] Event distribution: Events are sent to the corresponding shard topic partitions through Kafka Producer; subscriber groups (such as replica nodes across data centers) monitor their respective partitions.

[0140] Incremental data synchronization: By traversing every byte of two data blocks, the difference patch is generated using the characteristics of the XOR operation:

[0141] Initialize patch storage space: Create a bytearray object xor_result with the same length as src_data (or dst_data, because they have the same length) to store the final XOR result (i.e., differential patch).

[0142] Byte-by-byte XOR calculation: traverse the index i of each byte (from 0 to len(src_data)-1), perform an XOR operation (^) on src_data[i] and dst_data[i], and store the result in xor_result[i].

[0143] XOR operation characteristics: For binary bits, the result of the XOR operation is 1 if and only if the two input bits are different; if they are the same, the result is 0. Therefore, each position in xor_result that is 1 indicates that src_data and dst_data are different in the corresponding binary bit of that byte, and 0 indicates that they are the same.

[0144] Return the differential patch: Finally, xor_result is returned, whose content is the binary difference between src_data and dst_data.

[0145] RDMA zero-copy transmission: The local host submits a write request (including data address, length, remote memory address, and other information) through ibv_post_send.

[0146] The RDMA network card (HCA, Host Channel Adapter) takes the request from the send queue and transmits the data directly to the remote host through the InfiniBand / RoCE network.

[0147] After receiving the data, the HCA of the remote host directly writes it into the remote memory according to the target memory address in the request (pre-registration and authorization are required).

[0148] After the sending is completed, the local HCA generates a completion notification and puts it into the completion queue (CQ) for the application to query.

[0149] Synchronous execution optimization: Use RDMA Send-Recv primitives to transmit patch packets; support multi-threaded parallel patch application (number of threads = number of CPU cores / 2); update the local version number through Compare-And-Swap after application completion.

[0150] Exception handling and disaster recovery mechanism.

[0151] Shard synchronization failure handling: The Exponential Backoff algorithm is used to calculate the waiting time for retry operations (such as network requests and resource access). The core idea is to use exponentially increasing delay time to avoid server pressure or network congestion caused by a large number of retry requests in a short period of time. At the same time, random jitter is combined to further disperse the retry time and improve system robustness.

[0152] Exponential growth: 2^attempt is the core growth factor, ensuring that the delay of each retry increases exponentially (e.g., 100ms×1×jitter for attempt=0, 100ms×2×jitter for attempt=1, 100ms×4×jitter for attempt=2, and so on).

[0153] Random jitter: Multiply the delay by jitter (a random number between 0.5 and 1.5) to ensure that the actual delay fluctuates within the range of [base_delay × 2^attempt × 0.5, base_delay × 2^attempt × 1.5] to prevent complete synchronization of retry times among multiple clients.

[0154] Rounding: The result is converted into an integer (millisecond level) through the int() function to meet the actual requirements of time accuracy.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0156] The embodiment of the present application also provides a synchronization device for storing metadata. Figure 6 A schematic diagram of a structure of a synchronization device for storing metadata provided by an embodiment of the present disclosure is shown in FIG. Figure 6As shown, including:

[0157] A mapping unit 21 is configured to map data shards to target object storage device nodes based on a consistent hashing algorithm, and maintain a mapping relationship between the shards and the object storage device nodes through a monitoring node;

[0158] A sending unit 22 is configured to encapsulate the data change event into a message body and send the message to the partition corresponding to the target shard;

[0159] The receiving unit 23 is configured to pre-check received events based on the Bloom filter in the local cache, filter out invalid shard events, and only receive change events related to the local shard;

[0160] The transmission unit 24 is used to transmit the check value of the changed data block and pre-register the memory area.

[0161] Furthermore, in a possible implementation of the embodiment of the present disclosure, the mapping unit 21 is further configured to:

[0162] Calculate the hash value of the data block based on the preset algorithm, and determine the target shard through a consistent hash ring with the number of virtual nodes equal to the number of physical nodes;

[0163] Dynamically adjust the shard mapping strategy based on the storage type of the object storage device node.

[0164] Furthermore, in a possible implementation of the embodiment of the present disclosure, the sending unit 22 is further configured to:

[0165] The message body includes a fragment identifier, event type, version number and check value;

[0166] Events are sent to the partitions via an asynchronous publishing mechanism, where each subscriber group corresponds to at least one consumer node.

[0167] Furthermore, in a possible implementation of the embodiment of the present disclosure, the receiving unit 23 is further configured to:

[0168] The Bloom filter synchronizes the latest shard list based on a preset time interval.

[0169] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 7 As shown, the device also includes:

[0170] The sharding unit 25 is configured to perform shard synchronization by using an exponential backoff algorithm in response to a shard synchronization failure.

[0171] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 7 As shown, the device also includes:

[0172] The migration unit 26 is configured to trigger shard migration when the load of the object storage device node exceeds a preset threshold, a new node joins the cluster, or an existing node fails and goes offline.

[0173] Furthermore, in a possible implementation of the embodiment of the present disclosure, the transmission of the check value of the changed data block adopts an incremental synchronization protocol, and only the check value of the changed data block is transmitted to generate a differential patch.

[0174] For descriptions of features in the embodiments corresponding to the apparatus for synchronizing metadata, reference may be made to the relevant descriptions of the embodiments corresponding to the method for synchronizing metadata, which will not be described in detail here.

[0175] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps of any of the above-mentioned embodiments of the method for synchronizing stored metadata.

[0176] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the method for synchronizing stored metadata when running.

[0177] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0178] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned embodiments of the method for synchronizing stored metadata are implemented.

[0179] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned embodiments of the method for synchronizing the storage metadata are implemented.

[0180] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] The above is a detailed introduction to a synchronization method, device, electronic device and storage medium for storing metadata provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for synchronizing stored metadata, characterized in that: include: Map data shards to target object storage device nodes based on the consistent hashing algorithm, and maintain the mapping relationship between shards and object storage device nodes through monitoring nodes; Encapsulate the data change event as a message body and send it to the partition corresponding to the target shard; Pre-check received events against the locally cached Bloom filter, filter out invalid shard events, and only receive change events related to the local shard; Transmits the checksum of the changed data block and pre-registers the memory area.

2. The method for synchronizing storage metadata according to claim 1, wherein: Mapping the data shards to the target object storage device nodes based on the consistent hashing algorithm and maintaining the mapping relationship between the shards and the object storage device nodes through the monitoring node also includes: Calculate the hash value of the data block based on the preset algorithm, and determine the target shard through a consistent hash ring with the number of virtual nodes equal to the number of physical nodes; Dynamically adjust the shard mapping strategy based on the storage type of the object storage device node.

3. The method for synchronizing storage metadata according to claim 1, wherein: The data change event is encapsulated as a message body and sent to the partition corresponding to the target shard, including: The message body includes a fragment identifier, event type, version number and check value; Events are sent to the partitions via an asynchronous publishing mechanism, where each subscriber group corresponds to at least one consumer node.

4. The method for synchronizing storage metadata according to claim 1, wherein: Pre-checking received events based on the locally cached Bloom filter, filtering invalid shard events, and only receiving change events related to local shards further includes: The Bloom filter synchronizes the latest shard list based on a preset time interval.

5. The method for synchronizing storage metadata according to claim 1, wherein: The method further comprises: In response to a shard synchronization failure, an exponential backoff algorithm is employed to synchronize the shards.

6. The method for synchronizing storage metadata according to any one of claims 1 to 5, characterized in that: The method further comprises: Shard migration is triggered when the load of the object storage device node exceeds the preset threshold, a new node joins the cluster, or an existing node fails and goes offline.

7. The method for synchronizing storage metadata according to claim 1, wherein: The check value of the changed data block is transmitted using an incremental synchronization protocol, and only the check value of the changed data block is transmitted to generate a differential patch.

8. A synchronization device for storing metadata, characterized in that: include: A mapping unit, configured to map data shards to target object storage device nodes based on a consistent hashing algorithm, and maintain a mapping relationship between shards and object storage device nodes through a monitoring node; The sending unit is used to encapsulate the data change event into a message body and send it to the partition corresponding to the target shard; A receiving unit, configured to pre-check received events based on a locally cached Bloom filter, filter out invalid shard events, and only receive change events related to the local shard; The transmission unit is used to transmit the check value of the changed data block and pre-register the memory area.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for synchronizing stored metadata as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for synchronizing stored metadata according to any one of claims 1 to 7 are implemented.

Images