A data synchronization method, device, readable storage medium and program product

Abstract

The application discloses a data synchronization method and device, a readable storage medium and a program product, relates to the technical field of storage, and comprises a main and auxiliary metadata separation asynchronous architecture of a processor core, and lock operation most sensitive to performance is removed, so that metadata lock contention delay in a multi-core high-concurrency environment and performance jitter caused thereby are fundamentally eliminated. A hot storage block that causes greater pressure on a network and a backend storage can be accurately identified, a log pointer array is set to record changed data of the hot storage block, and the synchronization mode of the hot storage block is degraded from transmitting the entire storage block to transmitting only accurate changes, so that the technical problems of low resource utilization and resource waste caused by multiple cores contending for a bitmap lock are solved, the technical effects of eliminating metadata lock contention delay in a multi-core high-concurrency environment and performance jitter caused thereby, improving resource utilization and improving data synchronization efficiency are achieved.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a data synchronization method, device, readable storage medium, and program product. Background Technology

[0002] Current volume replication methods primarily involve logically dividing the entire source storage volume into a series of fixed-size storage blocks. The system maintains a bitmap in memory, where each bit corresponds one-to-one with a storage block on the volume. When a write input / output (I / O) request arrives, the system acquires a bitmap lock to access the bitmap, marking the storage block overwritten by the write request as dirty. Subsequently, the data in the dirty storage block is synchronized to the remote end, and the lock is released.

[0003] This can easily lead to write amplification issues and low resource utilization. In high-concurrency scenarios, a massive number of I / O requests will compete for the same bitmap lock, causing processor cores that fail to acquire the lock to waste resources. Summary of the Invention

[0004] This invention provides a data synchronization method, device, readable storage medium, and program product to at least solve the problems of low resource utilization and resource waste caused by multiple cores contending for bitmap locks in related technologies.

[0005] This invention provides a data synchronization method, comprising:

[0006] Within each preset time window, the write operations of each memory block are recorded using the private bitmap in the main metadata structure corresponding to each processor core.

[0007] The write counter array in each master metadata structure is used to record the number of writes to each storage block within each preset time window;

[0008] Based on the number of writes recorded in the write counter array, each dirty storage block with write operations is divided into cold storage blocks and hot storage blocks, and the log pointer array corresponding to each processor core is used to record the changed data of the hot storage blocks; wherein, the log pointer array includes the log address corresponding to the memory where the changed data of the hot storage blocks is stored;

[0009] Each private bitmap is aggregated according to a preset aggregation period to obtain a global dirty bitmap; wherein the preset aggregation period is greater than or equal to the preset time window;

[0010] Generate a global log association information table based on each log pointer array;

[0011] Data synchronization is performed based on the global dirty bitmap and the global log association information table.

[0012] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described data synchronization methods when executing the computer program.

[0013] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data synchronization methods.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data synchronization methods.

[0015] This invention, through its asynchronous architecture separating primary and secondary metadata of the processor cores, eliminates the most performance-sensitive lock operations, fundamentally eliminating metadata lock contention latency and the resulting performance jitter in multi-core high-concurrency environments. It can accurately identify hot storage blocks that put significant pressure on the network and backend storage, setting log pointer arrays to record changes to these blocks. This downgrades the synchronization mode of hot storage blocks from transmitting the entire block to transmitting only precise changes, significantly reducing network bandwidth and backend storage I / O load required for data synchronization, improving resource utilization and data synchronization efficiency. Therefore, it solves the technical problems of low resource utilization and resource waste caused by multiple cores contending for bitmap locks, achieving the technical effects of eliminating metadata lock contention latency and the resulting performance jitter in multi-core high-concurrency environments, improving resource utilization, and enhancing data synchronization efficiency. Attached Figure Description

[0016] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 An architecture diagram of a data synchronization system provided in an embodiment of the present invention;

[0018] Figure 2 A flowchart illustrating the implementation of a data synchronization method provided in an embodiment of the present invention;

[0019] Figure 3 A flowchart illustrating the implementation of another data synchronization method provided in this embodiment of the invention;

[0020] Figure 4 This is a structural block diagram of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0022] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] The specific application environment architecture or specific hardware architecture on which the execution of the data synchronization method depends is described here.

[0025] See Figure 1 , Figure 1 This is an architecture diagram of a data synchronization system provided in an embodiment of the present invention. The system architecture mainly consists of three modules: a lock-free tracking module, an adaptive block management module, and an aggregation synchronization module.

[0026] Lock-free tracing module: Located at the forefront, it directly faces and processes application I / O requests. Through an innovative design that separates primary and secondary metadata, it creates an exclusive primary metadata structure for each processor core, thereby enabling lock-free operations on change records.

[0027] Storage Block Management Module: Building upon the performance offered by the lock-free architecture, this module implements a strategy that dynamically adjusts the tracking granularity based on real-time workload. It dynamically and seamlessly switches frequently written hot storage blocks from the default coarse-grained bitmap tracking mode to a fine-grained log address tracking mode by analyzing the write patterns of storage blocks in real time.

[0028] The aggregation and synchronization module, acting as the system's backend, is handled by a separate, low-priority thread. It periodically and asynchronously aggregates all processor core-private change information securely into a global sub-bitmap, and ultimately, the data synchronization process performs efficient, differentiated intelligent synchronization based on the aggregated metadata.

[0029] The embodiments of the present invention provide a data synchronization method, and the method is described in detail in conjunction with the execution flow of the data synchronization method.

[0030] See Figure 2 , Figure 2 The following is a flowchart illustrating the implementation of a data synchronization method according to an embodiment of the present invention. The method may include the following steps.

[0031] S201: Within each preset time window, the write operations of each memory block are recorded using the private bitmap in the master metadata structure corresponding to each processor core.

[0032] A time window for data synchronization is pre-set, and a corresponding master metadata structure is pre-built for each processor core. Each master metadata structure contains a private bitmap, thus obtaining a private bitmap for each processor core. Within each preset time window, write operations on each storage block are recorded using the private bitmap in the master metadata structure corresponding to each processor core.

[0033] It should be noted that the preset time window can be set and adjusted according to the actual situation. The comparison of the embodiments of the present invention is not limited, such as it can be set to 10ms.

[0034] S202: Use the write counter array in each master metadata structure to record the number of writes to each storage block within each preset time window.

[0035] Each master metadata structure also includes a write counter array. Each element in the write counter array corresponds to a storage block and is used to count the number of times each storage block is written to by the processor core within a preset time window. The write counter array in each master metadata structure is used to record the number of writes to each storage block within each preset time window. By recording the number of writes to each storage block within each preset time window, the write frequency of each processor core to each storage block can be obtained.

[0036] S203: Based on the number of writes recorded in the write counter array, each dirty storage block with write operations is divided into cold storage blocks and hot storage blocks, and the change data of the hot storage blocks is recorded using the log pointer array corresponding to each processor core.

[0037] The log pointer array includes the log addresses corresponding to the memory locations where changed data for hot storage blocks are stored.

[0038] A log pointer array is pre-configured for each processor core, containing log addresses corresponding to the memory used to store changed data in hot storage blocks. Storage blocks with data writes within a preset time window are designated as dirty storage blocks. After recording the write counts for each storage block within each preset time window using the write counter array in each master metadata structure, the dirty storage blocks with write operations are divided into cold and hot storage blocks based on the write counts recorded in the write counter array. Change data is then recorded for the hot storage blocks using the log pointer array corresponding to each processor core. By dividing hot and cold storage blocks according to the number of writes and recording change data for hot storage blocks, subsequent data synchronization of hot storage blocks only synchronizes the changed data, eliminating the need to synchronize the entire block's data, significantly reducing the amount of data synchronized and improving data synchronization efficiency.

[0039] S204: Aggregate each private bitmap according to the preset aggregation cycle to obtain the global dirty bitmap.

[0040] Among them, the preset aggregation period is greater than or equal to the preset time window.

[0041] A preset aggregation period is set, which is greater than or equal to a preset time window. After recording the write operations of each storage block using the private bitmap in the main metadata structure corresponding to each processor core, the private bitmaps are aggregated according to the preset aggregation period to obtain a global dirty bitmap. This summarizes the storage blocks that have write operations within the preset time window.

[0042] The preset aggregation period can be set to an integer multiple of the preset time window. For example, when the preset time window is set to 10ms, the preset aggregation period can be set to 1s. By setting the preset aggregation period to an integer multiple of the preset time window, the integrity of the aggregated data is ensured.

[0043] S205: Generate a global log association information table based on each log pointer array.

[0044] After recording the number of writes to each storage block within each preset time window using the write counter array in each master metadata structure, a global log association information table is generated based on each log pointer array. By merging the changed data recorded in the log pointer array, the global log association information table is obtained, thereby summarizing the data changes of the hot storage block within the entire preset time window.

[0045] S206: Synchronize data based on the global dirty bitmap and the global log association information table.

[0046] After aggregating each private bitmap at the end of each preset time window to obtain a global dirty bitmap, and generating a global log association information table based on each log pointer array, data synchronization is performed based on the global dirty bitmap and the global log association information table. By combining the global dirty bitmap and the global log association information table, data in cold storage blocks where no changed data records exist in the global log association information table can be synchronized as a whole. For hot storage blocks where changed data records exist in the global log association information table, only the changed data is synchronized. This achieves differentiated synchronization of cold and hot storage blocks, significantly reducing the amount of data synchronization and improving data synchronization efficiency.

[0047] This invention, through its asynchronous architecture separating primary and secondary metadata of the processor cores, eliminates the most performance-sensitive lock operations, fundamentally eliminating metadata lock contention latency and the resulting performance jitter in multi-core high-concurrency environments. It can accurately identify hot storage blocks that put significant pressure on the network and backend storage, setting log pointer arrays to record changes to these blocks. This downgrades the synchronization mode of hot storage blocks from transmitting the entire block to transmitting only precise changes, significantly reducing network bandwidth and backend storage I / O load required for data synchronization, improving resource utilization and data synchronization efficiency. Therefore, it solves the technical problems of low resource utilization and resource waste caused by multiple cores contending for bitmap locks, achieving the technical effects of eliminating metadata lock contention latency and the resulting performance jitter in multi-core high-concurrency environments, improving resource utilization, and enhancing data synchronization efficiency.

[0048] See Figure 3 , Figure 3 The following is a flowchart illustrating another data synchronization method provided in an embodiment of the present invention. The method may include the following steps.

[0049] S301: Within each preset time window, the write operations of each memory block are recorded using the private bitmap in the master metadata structure corresponding to each processor core.

[0050] S302: Get the preset write count threshold.

[0051] Set a threshold for the number of writes within a preset time window, and then obtain the preset threshold for the number of writes.

[0052] It should be noted that the preset write count threshold can be set and adjusted according to the actual situation. The comparison of the embodiments of the present invention is not limited, such as it can be set to 3 times.

[0053] In one specific embodiment of the present invention, step S302 may include the following steps:

[0054] Step 1: Obtain the memory usage rate corresponding to the start point of each preset time window;

[0055] Step 2: Determine the preset write count threshold for each preset time window based on memory usage.

[0056] For ease of description, the two steps above can be combined for explanation.

[0057] The write count threshold can be dynamically and adaptively set based on memory usage. The memory usage rate corresponding to the start point of each preset time window is obtained, and the preset write count threshold for each preset time window is determined based on the memory usage rate. For example, when system memory resources are scarce, the preset write count threshold can be automatically increased, making the policy upgrade conditions more stringent, thereby reducing log address allocation and protecting system memory. By dynamically and adaptively setting the write count threshold based on memory usage, system memory is utilized more efficiently.

[0058] S303: Divide each dirty storage block into cold storage blocks and hot storage blocks according to the preset write count threshold and the write count recorded in the write counter array, and determine whether there is a log address corresponding to the hot storage block in the log pointer array. If not, proceed to step S304; if yes, proceed to step S305.

[0059] After obtaining the preset write count threshold, each dirty storage block is divided into cold storage blocks and hot storage blocks based on the preset write count threshold and the write count recorded in the write counter array. Specifically, the preset write count threshold can be used as the dividing line between cold and hot storage blocks. Storage blocks with write counts greater than or equal to the preset write count threshold are identified as hot storage blocks, and storage blocks with write counts less than the preset write count threshold are identified as cold storage blocks. After identifying the hot storage blocks, it is checked whether there is a log address corresponding to the hot storage block in the log pointer array. If not, it means that the hot storage block is being identified as a hot storage block for the first time within the preset time window, and step S304 is executed. If yes, it means that the hot storage block is not being identified as a hot storage block for the first time within the preset time window, and step S305 is executed.

[0060] In one specific embodiment of the present invention, dividing each dirty storage block into cold storage blocks and hot storage blocks according to a preset write count threshold and the write count recorded in the write counter array may include the following steps:

[0061] Step 1: Compare the number of writes corresponding to each dirty storage block recorded in the write counter array with the preset write count threshold;

[0062] Step 2: Identify dirty storage blocks that have a write count greater than or equal to a preset write count threshold as hot storage blocks;

[0063] Step 3: Identify the dirty storage blocks corresponding to the number of writes less than the preset write count threshold as cold storage blocks.

[0064] For ease of description, the three steps above can be combined for explanation.

[0065] After obtaining the preset write count threshold, the write counts corresponding to each dirty storage block recorded in the write counter array are compared with the preset write count threshold. Dirty storage blocks with write counts greater than or equal to the preset write count threshold are identified as hot storage blocks, and dirty storage blocks with write counts less than the preset write count threshold are identified as cold storage blocks. By setting the preset write count threshold as the dividing line between hot and cold storage blocks, fast and accurate division of hot and cold storage blocks is achieved.

[0066] In one specific embodiment of the present invention, dividing each dirty storage block into cold storage blocks and hot storage blocks according to a preset write count threshold and the write count recorded in the write counter array may include the following steps:

[0067] The hysteresis control mechanism divides each dirty storage block into cold storage blocks and hot storage blocks based on a preset write count threshold and the write count recorded in the write counter array.

[0068] After obtaining the preset write count threshold, a hysteresis control mechanism divides each dirty storage block into cold and hot storage blocks based on the preset write count threshold and the write count recorded in the write counter array. For example, if the time window is set to 10ms and the write count threshold is 3 times, when a storage block is first identified as a hot storage block, it will only be identified as a hot storage block and assigned a log address if the write frequency of a storage block exceeds a higher hot threshold (Threshold_Hot) within the time window, such as more than 5 writes within 10ms. A storage block that has been marked as a hot storage block will only have its log address reclaimed and be downgraded to a cold storage block in subsequent background cleanup if its write frequency remains below a lower cold threshold (Threshold_Cold) for a longer time window, such as less than 1 write per second. A stable buffer is formed between Threshold_Hot and Threshold_Cold, which prevents storage blocks whose write behavior is in a critical state (i.e., storage blocks whose write frequency fluctuates around the preset write count threshold) from frequently switching between cold and hot storage blocks. This avoids the log address being repeatedly allocated and released, and greatly enhances the stability of cold and hot storage block switching.

[0069] S304: Determine that the hot storage block is the first hot storage block identified within the preset time window, allocate a log address to the hot storage block, store the allocated log address in the log pointer array, and use the memory corresponding to the log address to record the changed data.

[0070] If the log pointer array does not contain a log address corresponding to the hot storage block, the hot storage block is determined to be the first hot storage block identified within a preset time window. Since the log pointer array corresponding to this processor core does not yet contain a log address corresponding to this hot storage block within the same preset time window, a log address is allocated to the hot storage block, and the allocated log address is stored in the log pointer array. The memory corresponding to the log address is used to record the changed data. By allocating log addresses to the storage blocks of the first identified hot storage blocks, accurate recording of changes to the hot storage blocks is ensured.

[0071] S305: Determine that the hot storage block is not the first time it has been identified as a hot storage block within the preset time window, find the log address corresponding to the hot storage block from the log pointer array, obtain the memory offset corresponding to the log address, and add the changed data to the memory corresponding to the log address according to the offset.

[0072] The log pointer array includes the log addresses corresponding to the memory locations where changed data for hot storage blocks are stored.

[0073] If the log pointer array contains a log address corresponding to a hot storage block, the hot storage block is determined to be one that has been identified as a hot storage block before within the preset time window. The log address corresponding to the hot storage block is then searched in the log pointer array, and the memory offset corresponding to that log address is obtained. Based on this offset, the changed data is added to the memory corresponding to the log address. By directly adding the changed data corresponding to the hot storage block whose log address exists in the log pointer array to the memory corresponding to that log address, this append-only writing method transforms multiple random write operations to the same hot storage block into a single, efficient sequential write operation in memory. This achieves accurate and orderly recording of changed data in the hot storage block, providing convenient data retrieval for subsequent data synchronization and further improving data synchronization efficiency. Furthermore, it avoids redundant memory allocation and enables rapid recording of changed data.

[0074] S306: Acquire the lock of the global sub-bitmap according to the preset aggregation cycle, and aggregate each private bitmap by traversing the main metadata structure corresponding to each processor core to obtain the global dirty bitmap.

[0075] After the change data of the hot storage block is recorded, the lock of the global sub-bitmap is acquired according to the preset aggregation cycle. The private bitmaps are aggregated by traversing the main metadata structure corresponding to each processor core to obtain the global dirty bitmap.

[0076] S307: Release the lock on the global sub-bitmap.

[0077] After aggregating to obtain the global dirty bitmap, the lock on the global sub-bitmap is released. By acquiring the lock on the global sub-bitmap when aggregating each private bitmap at the end of each preset time window, the global sub-bitmap is locked, avoiding interference from write operations of each storage block in the next aggregation cycle, and ensuring accurate recording of write operations of each storage block within each preset time window. This changes the granularity of locking from the traditional method of locking once per write request to locking once per aggregation cycle in this embodiment of the invention, avoiding frequent use of locks, ensuring that the system can still provide low-latency, high-throughput data synchronization capabilities under high load, and improving data synchronization efficiency.

[0078] S308: Generate a global log association information table based on each log pointer array.

[0079] S309: Generate a global sub-map based on the global dirty bitmap and the global log association information table.

[0080] After aggregating to obtain the global dirty bitmap and generating a global log association information table based on each log pointer array, a global sub-bitmap is generated based on the global dirty bitmap and the global log association information table, thus obtaining the global sub-bitmap corresponding to this preset time window. The global sub-bitmap enables comprehensive and accurate recording of write operations on each storage block within this preset time window, providing accurate reference data for data synchronization.

[0081] S310: Determine the dirty storage blocks with write operations within each preset time window based on the global sub-bitmap, and obtain the block identifier of the dirty storage block.

[0082] After generating a global sub-bitmap based on the global dirty bitmap and the global log association information table, the global sub-bitmap records the storage number of each storage block and the write status of each storage block. Based on the global sub-bitmap, dirty storage blocks with write operations within each preset time window are identified, and the block identifier of the dirty storage block is obtained.

[0083] S311: Perform log address lookup in the global log association information table based on the block identifier.

[0084] For dirty storage blocks with log addresses, the correspondence between the block identifier of the dirty storage block and the log address is pre-stored in the global log association information table. After obtaining the block identifier of the dirty storage block, the log address is looked up in the global log association information table based on the block identifier.

[0085] S312: For the data in the dirty storage block where the log address is found, synchronize the data according to the changed data recorded in the memory corresponding to the log address.

[0086] By looking up the log address in the global log association information table based on the block identifier, if the log address is found in the global log association information table, it indicates that the dirty storage block is a hot storage block. For the data in the dirty storage block where the log address is found, data synchronization is performed according to the changed data recorded in memory corresponding to the log address.

[0087] S313: For data in a dirty storage block where the log address cannot be found, synchronize all data in the dirty storage block.

[0088] By searching the global log association information table based on the block identifier, if no log address is found in the global log association information table, it is indicated that the dirty storage block is a cold storage block. For the data in the dirty storage block where no log address is found, all data in the dirty storage block is synchronized.

[0089] By looking up the log address in the global log association information table and dividing the dirty storage block into hot and cold storage blocks based on the lookup results, it is possible to quickly find the changed data in the hot storage block. Then, synchronization is performed according to the data synchronization methods corresponding to the hot and cold storage blocks respectively. By synchronizing only the changed data in the hot storage block, the data synchronization efficiency is improved.

[0090] In this embodiment of the invention, the entire strategy switching process from coarse-grained bitmap tracking to fine-grained log address tracking is automatically triggered and managed by the system based on real-time workload, without any manual intervention, parameter configuration, or post-tuning, which greatly improves versatility.

[0091] In one specific embodiment of the present invention, before recording the write operations of each memory block using the private bitmap in the main metadata structure corresponding to each processor core within each preset time window, the method may further include the following steps:

[0092] Step 1: Obtain the non-uniform memory access node to which each processor core belongs;

[0093] Step 2: Allocate memory from the non-consistent memory access nodes to which each processor core belongs, and use the allocated memory to build the master metadata structure corresponding to each processor core.

[0094] For ease of description, the two steps above can be combined for explanation.

[0095] The algorithm retrieves the Non-Uniform Memory Access (NUMA) nodes belonging to each processor core, allocates memory from these nodes, and uses the allocated memory to construct the master metadata structure for each processor core. By allocating memory from the NUMA nodes to construct the master metadata structure, the NUMA affinity of the constructed master metadata structure is improved.

[0096] In one specific embodiment of the present invention, recording the write operations of each memory block using the private bitmap in the main metadata structure corresponding to each processor core may include the following steps:

[0097] Step 1: Determine the logical address and data length corresponding to each received write request;

[0098] Step 2: Determine the storage block corresponding to each write request based on the logical address and data length;

[0099] Step 3: Determine the processor core corresponding to each write request, and use the processor core to mark the marker position of the storage block in the private bitmap as dirty, thus obtaining each dirty data block.

[0100] For ease of description, the three steps above can be combined for explanation.

[0101] Upon receiving each write request, the system determines the corresponding logical address and data length for each request. Based on these parameters, it identifies the memory block and processor core associated with each write request. The processor core is then used to mark the memory block's flag in the private bitmap as dirty, thus obtaining the dirty data blocks. By calculating the memory blocks covered by each write request based on the logical address and data length, and identifying these covered blocks as dirty memory blocks, the system achieves fast and accurate identification of dirty memory blocks.

[0102] In one specific embodiment of the present invention, the method may further include the following steps:

[0103] After aggregating the private bitmaps in each master metadata structure into the global sub-bitmap, clear each master metadata structure.

[0104] After aggregating the private bitmaps in each primary metadata structure into the global secondary bitmap, each primary metadata structure is cleared. By promptly clearing each primary metadata structure after aggregating the private bitmaps into the global secondary bitmap, the primary metadata structures are released in a timely manner, avoiding interference with storage block write operation records in the next aggregation cycle.

[0105] In one specific embodiment of the present invention, generating a global log association information table based on each log pointer array may include the following steps:

[0106] Step 1: Determine the storage block containing the log address based on each log pointer array;

[0107] Step 2: Insert the log address into the global log association information table, and mark the log address flag corresponding to the storage block containing the log address as having a log address in the global sub-bitmap; the global sub-bitmap is constructed from the global dirty bitmap and the global log association information table.

[0108] For ease of description, the two steps above can be combined for explanation.

[0109] After obtaining the global dirty bitmap and the global log association information table, the storage blocks containing log addresses are determined based on the log pointer arrays. These log addresses are then inserted into the global log association information table, and the log address flag corresponding to the storage block containing the log address is set to the "log address present" state in the global sub-bitmap. By recording the log address status of each storage block in the global sub-bitmap, accurate indications are provided for log address lookup during subsequent data synchronization, further improving data synchronization efficiency.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0111] Embodiments of the present invention also provide a data synchronization device.

[0112] See Figure 4 , Figure 4 This is a structural block diagram of a data synchronization device provided in an embodiment of the present invention. The device may include:

[0113] The write operation recording module 41 is used to record the write operations of each storage block in each preset time window using the private bitmap in the main metadata structure corresponding to each processor core.

[0114] The write count recording module 42 is used to record the number of writes to each storage block within each preset time window using the write counter array in each master metadata structure;

[0115] The change data recording module 43 is used to divide each dirty storage block with write operations into cold storage blocks and hot storage blocks according to the number of writes recorded in the write counter array, and to record change data for the hot storage blocks using the log pointer array corresponding to each processor core; wherein, the log pointer array includes the log address corresponding to the memory where the change data of the hot storage block is stored;

[0116] The global dirty bitmap acquisition module 44 is used to aggregate each private bitmap at the end of each preset time window to obtain the global dirty bitmap.

[0117] The association information table generation module 45 is used to generate a global log association information table based on each log pointer array;

[0118] The data synchronization module 46 is used to synchronize data based on the global dirty bitmap and the global log association information table.

[0119] This invention, through its asynchronous architecture separating primary and secondary metadata of the processor cores, eliminates the most performance-sensitive lock operations, fundamentally eliminating metadata lock contention latency and the resulting performance jitter in multi-core high-concurrency environments. It can accurately identify hot storage blocks that put significant pressure on the network and backend storage, setting log pointer arrays to record changes to these blocks. This downgrades the synchronization mode of hot storage blocks from transmitting the entire block to transmitting only precise changes, significantly reducing network bandwidth and backend storage I / O load required for data synchronization, improving resource utilization and data synchronization efficiency. Therefore, it solves the technical problems of low resource utilization and resource waste caused by multiple cores contending for bitmap locks, achieving the technical effects of eliminating metadata lock contention latency and the resulting performance jitter in multi-core high-concurrency environments, improving resource utilization, and enhancing data synchronization efficiency.

[0120] In one specific embodiment of the present invention, the data synchronization module 46 may include:

[0121] The global sub-bitmap generation submodule is used to generate a global sub-bitmap based on the global dirty bitmap and the global log association information table;

[0122] The block identifier acquisition submodule is used to determine the dirty storage blocks that have write operations within each preset time window based on the global sub-bitmap, and to obtain the block identifier of the dirty storage blocks;

[0123] The log address lookup submodule is used to look up the log address in the global log association information table based on the block identifier.

[0124] The first data synchronization submodule is used to synchronize the data in the dirty storage block where the log address is found with the changed data recorded in the memory corresponding to the log address.

[0125] The second data synchronization submodule is used to synchronize all data in dirty storage blocks where the log address cannot be found.

[0126] In one specific embodiment of the present invention, the device may further include:

[0127] The node acquisition module is used to acquire the non-consistent memory access node to which each processor core belongs before recording the write operation of each storage block using the private bitmap in the main metadata structure corresponding to each processor core within each preset time window.

[0128] The master metadata structure construction module is used to allocate memory from the non-consistent memory access nodes to which each processor core belongs, and to use the allocated memory to construct the master metadata structure corresponding to each processor core.

[0129] In one specific embodiment of the present invention, the change data recording module 43 may include:

[0130] The threshold acquisition submodule is used to obtain the preset write count threshold;

[0131] The cold and hot storage block partitioning submodule is used to partition each dirty storage block into cold storage blocks and hot storage blocks based on a preset write count threshold and the write count recorded in the write counter array.

[0132] In one specific embodiment of the present invention, the cold and hot storage block partitioning submodule may include:

[0133] The comparison unit is used to compare the number of writes corresponding to each dirty storage block recorded in the write counter array with the preset write count threshold.

[0134] A hot storage block determination unit is used to determine dirty storage blocks corresponding to write counts greater than or equal to a preset write count threshold as hot storage blocks.

[0135] The cold storage block determination unit is used to determine the dirty storage blocks corresponding to a number of writes less than a preset write count threshold as cold storage blocks.

[0136] In one specific embodiment of the present invention, the threshold acquisition submodule may include:

[0137] The memory usage acquisition unit is used to acquire the memory usage corresponding to the start point of each preset time window.

[0138] The threshold determination unit is used to determine the preset write count threshold corresponding to each preset time window based on the memory usage rate.

[0139] In one specific embodiment of the present invention, the cold and hot storage block partitioning submodule is specifically a module that divides each dirty storage block into cold storage blocks and hot storage blocks according to a preset write count threshold and the write count recorded in the write counter array through a hysteresis control mechanism.

[0140] In one specific embodiment of the present invention, the write operation record module 42 may include:

[0141] The address and length determination submodule is used to determine the logical address and data length corresponding to each received write request.

[0142] The storage block determination submodule is used to determine the storage block corresponding to each write request based on the write logical address and data length.

[0143] The dirty data block acquisition submodule is used to determine the processor core corresponding to each write request, and use the processor core to set the flag position of the storage block in the private bitmap to dirty state, thereby obtaining each dirty data block.

[0144] In one specific embodiment of the present invention, the global dirty bitmap acquisition module 44 may include:

[0145] The global dirty bitmap acquisition submodule is used to acquire the lock of the global sub-bitmap according to the preset aggregation period. It aggregates each private bitmap by traversing the main metadata structure corresponding to each processor core to obtain the global dirty bitmap.

[0146] The lock release submodule is used to release the lock on the global sub-bitmap.

[0147] In one specific embodiment of the present invention, the device may further include:

[0148] The main metadata structure clearing module is used to clear each main metadata structure after aggregating the private bitmaps in each main metadata structure into the global sub-bitmap.

[0149] In one specific embodiment of the present invention, the change data recording module 43 may include:

[0150] The judgment submodule is used to determine whether the log pointer array contains a log address corresponding to a hot storage block;

[0151] The change data recording submodule is used to determine that the hot storage block is the first hot storage block identified within a preset time window when there is no log address corresponding to the hot storage block in the log pointer array. It then allocates a log address to the hot storage block, stores the allocated log address in the log pointer array, and uses the memory corresponding to the log address to record change data.

[0152] The Change Data Addition Submodule is used to determine if a hot storage block is identified as such within a preset time window when the log pointer array contains a log address corresponding to the hot storage block. It then searches for the log address corresponding to the hot storage block in the log pointer array, obtains the memory offset corresponding to the log address, and adds the change data to the memory corresponding to the log address based on the offset.

[0153] In one specific embodiment of the present invention, the association information table generation module 45 may include:

[0154] The storage block determination submodule is used to determine the storage block containing the log address based on each log pointer array;

[0155] The flag position submodule is used to insert the log address into the global log association information table and mark the log address corresponding to the storage block with the log address as having a log address in the global sub-bitmap; the global sub-bitmap is constructed from the global dirty bitmap and the global log association information table.

[0156] For a description of the features in the embodiment corresponding to the data synchronization device, please refer to the relevant description in the embodiment corresponding to the data synchronization method, which will not be repeated here.

[0157] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data synchronization method embodiments.

[0158] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described data synchronization method embodiments at runtime.

[0159] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0160] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data synchronization method embodiments.

[0161] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data synchronization method embodiments.

[0162] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0163] The data synchronization method, device, readable storage medium, and program product provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A data synchronization method, characterized in that, include: Within each preset time window, the write operations of each memory block are recorded using the private bitmap in the main metadata structure corresponding to each processor core. The write counter array in each master metadata structure is used to record the number of writes to each storage block within each preset time window; Based on the number of writes recorded in the write counter array, each dirty storage block with write operations is divided into cold storage blocks and hot storage blocks, and the log pointer array corresponding to each processor core is used to record the changed data of the hot storage blocks; wherein, the log pointer array includes the log address corresponding to the memory where the changed data of the hot storage blocks is stored; Each private bitmap is aggregated according to a preset aggregation period to obtain a global dirty bitmap; wherein the preset aggregation period is greater than or equal to the preset time window; Generate a global log association information table based on each log pointer array; Data synchronization is performed based on the global dirty bitmap and the global log association information table; The data synchronization based on the global dirty bitmap and the global log association information table includes: Generate a global sub-map based on the global dirty bitmap and the global log association information table; Based on the global sub-bitmap, determine the dirty storage blocks that have write operations within each preset time window, and obtain the block identifier of the dirty storage block; The log address is looked up in the global log association information table based on the block identifier; For the data in the dirty storage block where the log address is found, the data is synchronized according to the changed data recorded in memory corresponding to the log address; For data in dirty storage blocks where no log address is found, synchronize all data in the dirty storage blocks.

2. The data synchronization method according to claim 1, characterized in that, Before recording the write operations of each memory block using the private bitmap in the main metadata structure corresponding to each processor core within each preset time window, the process also includes: Obtain the non-consistent memory access node to which each processor core belongs; Memory is allocated from the non-uniform memory access nodes to which each processor core belongs, and the allocated memory is used to construct the master metadata structure corresponding to each processor core.

3. The data synchronization method according to claim 1, characterized in that, Based on the number of writes recorded in the write counter array, each dirty storage block with write operations is divided into cold storage blocks and hot storage blocks, including: Get the preset write count threshold; Based on the preset write count threshold and the write count recorded in the write counter array, each dirty storage block is divided into a cold storage block and a hot storage block.

4. The data synchronization method according to claim 3, characterized in that, Based on the preset write count threshold and the write count recorded in the write counter array, each dirty storage block is divided into cold storage blocks and hot storage blocks, including: The write counts corresponding to each dirty storage block recorded in the write counter array are compared with the preset write count threshold. Dirty storage blocks corresponding to write counts greater than or equal to the preset write count threshold are identified as hot storage blocks; Dirty storage blocks corresponding to write counts less than the preset write count threshold are identified as cold storage blocks.

5. The data synchronization method according to claim 3, characterized in that, Get the preset write count threshold, including: Obtain the memory usage rate corresponding to the start point of each preset time window; The preset write count thresholds corresponding to each preset time window are determined based on the memory usage rate.

6. The data synchronization method according to claim 3, characterized in that, Based on the preset write count threshold and the write count recorded in the write counter array, each dirty storage block is divided into cold storage blocks and hot storage blocks, including: The hysteresis control mechanism divides each dirty storage block into cold storage blocks and hot storage blocks based on the preset write count threshold and the write count recorded in the write counter array.

7. The data synchronization method according to claim 1, characterized in that, Write operations to each memory block are recorded using the private bitmap in the main metadata structure corresponding to each processor core, including: Determine the logical address and data length corresponding to each received write request; Based on the write logical address and the data length, determine the storage block corresponding to each write request; The processor core corresponding to each write request is determined, and the processor core is used to set the flag position of the storage block in the private bitmap to a dirty state to obtain each dirty data block.

8. The data synchronization method according to claim 1, characterized in that, The private bitmaps are aggregated according to a preset aggregation cycle to obtain a global dirty bitmap, including: The lock of the global sub-bitmap is acquired according to the preset aggregation cycle. The private bitmaps are aggregated by traversing the main metadata structure corresponding to each processor core to obtain the global dirty bitmap. Release the lock on the global sub-bitmap.

9. The data synchronization method according to claim 8, characterized in that, Also includes: After aggregating the private bitmaps in each master metadata structure into the global sub-bitmap, clear each master metadata structure.

10. The data synchronization method according to any one of claims 1 to 9, characterized in that, The hot storage block is modified using the log pointer array corresponding to each processor core, including: Determine whether the log pointer array contains the log address corresponding to the hot storage block; If not, then the hot storage block is determined to be the first hot storage block within the preset time window, a log address is allocated to the hot storage block, and the allocated log address is stored in the log pointer array. The memory corresponding to the log address is used to record the changed data. If so, the hot storage block is determined to be a hot storage block that is not the first time it has been determined within the preset time window. The log address corresponding to the hot storage block is found in the log pointer array, the offset of the memory corresponding to the log address is obtained, and the changed data is added to the memory corresponding to the log address according to the offset.

11. The data synchronization method according to claim 1, characterized in that, A global log association information table is generated based on each log pointer array, including: The storage block containing the log address is determined based on each log pointer array; The log address is inserted into the global log association information table, and the log address flag corresponding to the storage block containing the log address is set to the state of having a log address in the global sub-bitmap; wherein, the global sub-bitmap is constructed from the global dirty bitmap and the global log association information table.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data synchronization method as described in any one of claims 1 to 11 when executing the computer program.

Images