Data synchronization method, device, electronic device and computer-readable storage medium

Abstract

The present application discloses a data synchronization method, apparatus, electronic device, and computer-readable storage medium, relating to the field of distributed storage technology, including: detecting the positional relationship between a client and an object storage device; when the positional relationship indicates that the client and the object storage device are located on the same node, creating a shared memory channel, and synchronizing data between the client and the object storage device through the shared memory channel; when the positional relationship indicates that the client and the object storage device are located on different nodes, configuring an adaptive data transmission channel according to network performance parameters of a physical server, and synchronizing data between the client and the object storage device through the adaptive data transmission channel, wherein both the client and the object storage device are located on a physical server, solving the technical problem of how to apply NVMe‑oF to a distributed storage system to improve data synchronization efficiency.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data synchronization method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In distributed storage systems, data replica consistency synchronization in erasure coded pools is a key technology. Existing distributed storage systems rely on the TCP / IP network protocol for data synchronization. Although this technology is quite mature, traditional erasure coded pools have many shortcomings in replica consistency synchronization, such as high network overhead during synchronization, high intra-node transmission latency, and the inability to adaptively adjust TCP parameters, which severely restrict system performance. Against this backdrop, relevant fields have explored the application of NVMe-oF technology to erasure replica consistency in distributed storage systems. However, this technical direction faces challenges at multiple levels, including protocol adaptation, performance optimization, and system compatibility. Therefore, the technical challenge of applying NVMe-oF to distributed storage systems to improve data synchronization efficiency remains.

[0003] There is currently no effective solution to this technical problem.

[0004] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the Invention

[0005] The present application provides a data synchronization method, device, electronic device and computer-readable storage medium to at least solve the technical problem in the related art of how to apply NVMe-oF to a distributed storage system to improve data synchronization efficiency.

[0006] The present application provides a data synchronization method, comprising: detecting a positional relationship between a client and an object storage device; creating a shared memory channel when the positional relationship indicates that the client and the object storage device are located on the same node, and synchronizing data between the client and the object storage device through the shared memory channel; configuring an adaptive data transmission channel according to network performance parameters of a physical server when the positional relationship indicates that the client and the object storage device are located on different nodes, and synchronizing data between the client and the object storage device through the adaptive data transmission channel, wherein both the client and the object storage device are located on the physical server.

[0007] The present application also provides a data synchronization device, including: a detection module for detecting the positional relationship between a client and an object storage device; a first synchronization module for creating a shared memory channel when the positional relationship indicates that the client and the object storage device are located on the same node, and synchronizing data between the client and the object storage device through the shared memory channel; a second synchronization module for configuring an adaptive data transmission channel according to network performance parameters of a physical server when the positional relationship indicates that the client and the object storage device are located on different nodes, and synchronizing data between the client and the object storage device through the adaptive data transmission channel, wherein the client and the object storage device are both located on the physical server.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data synchronization methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data synchronization methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data synchronization methods when executed by a processor.

[0011] Through this application, the positional relationship between the client and the object storage device is intelligently detected, and the optimal data synchronization path is dynamically selected, which significantly improves the data synchronization efficiency. For example, when the client and the object storage device are in the same physical server node, the low latency characteristics of the shared memory channel can be used to exchange data directly in the memory without passing through the network stack, thereby greatly reducing the data synchronization delay and improving the throughput. Between different nodes, the adaptive data transmission channel can adjust the transmission strategy according to the real-time network conditions, such as dynamically adjusting the queue depth, transmission block size and timeout period to cope with network fluctuations and ensure the stability and efficiency of data synchronization. This method can solve the technical problem of how to apply NVMe-oF to distributed storage systems to improve data synchronization efficiency. Compared with the shortcomings of traditional synchronization methods with high latency within the same node and low efficiency across nodes, this application further reduces unnecessary network transmission, reduces the overall operating costs of the data center, and improves data synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of a distributed storage system architecture provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application;

[0016] Figure 4 4 is a block diagram of the computer system structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] Some technical terms in this application are explained.

[0021] NGN, Next Generation Network, next generation network.

[0022] NVMe-oF (Non-Volatile Memory Express over Fabrics) is a fabrics-based non-volatile memory block expression protocol, also known as the fabrics-based non-volatile memory host controller interface specification extension protocol. It enables high-speed access to remote NVMe storage devices. NVMe-oF allows NVMe devices to connect to servers over networks such as Ethernet, Fibre Channel, or InfiniBand. The main advantage of NVMe-oF is that it provides performance and low latency similar to local NVMe SSDs while enabling storage resources to be shared across multiple nodes.

[0023] In traditional data center architectures, storage devices are typically directly connected to a single server, limiting resource sharing and scalability. NVMe-oF, by leveraging RDMA (Remote Direct Memory Access) or other similar technologies, enables direct access to NVMe storage between different servers, significantly improving data transfer speeds and storage efficiency. This mechanism is ideal for applications requiring high bandwidth and low latency, such as high-performance computing, databases, artificial intelligence, and machine learning.

[0024] NVMe-oF supports a variety of different network transmission protocols, such as RoCE (full name RDMA over Converged Ethernet, remote direct memory access based on converged Ethernet), which uses Ethernet as the transmission medium but provides lower latency and higher performance than standard TCP / IP. It also supports iWARP (full name Internet Wide Area RDMA Protocol, Internet Wide Area Remote Direct Memory Access Protocol), which is an Ethernet-based RDMA protocol designed to improve the efficiency of remote access. With NVMe-oF, organizations can build larger-scale storage systems without sacrificing performance, while also simplifying management and improving resource utilization. This is especially important for building modern data centers and cloud infrastructures, as they need to process large amounts of data and support highly concurrent access.

[0025] OSD, Object Storage Daemon, object storage device, is responsible for storing data in the form of objects on the physical disk of each node in the cluster.

[0026] CRUSH, Controlled Replication Under Scalable Hashing, a controllable replication hash algorithm, is used for automatic sharding and replica placement of data in distributed storage.

[0027] MDS, Metadata Server, is used to manage the metadata of the file system.

[0028] BDP, Bandwidth-Delay Product, is equal to the product of network bandwidth and round-trip time, and is used to measure the capacity of a network link.

[0029] The embodiment of the present application provides a data synchronization method. In one embodiment, combined with Figure 1 The execution process of the data synchronization method is described in detail, including:

[0030] Step S102, detecting the position relationship between the client and the object storage device;

[0031] Step S104: if the location relationship indicates that the client and the object storage device are located on the same node, create a shared memory channel, and synchronize data between the client and the object storage device through the shared memory channel;

[0032] Step S106, when the position relationship indicates that the client and the object storage device are located at different nodes, configure an adaptive data transmission channel according to the network performance parameters of the physical server, and synchronize data between the client and the object storage device through the adaptive data transmission channel, wherein the client and the object storage device are both located on the physical server.

[0033] Through this application, the positional relationship between the client and the object storage device is intelligently detected, and the optimal data synchronization path is dynamically selected, which significantly improves the data synchronization efficiency. For example, when the client and the object storage device are in the same physical server node, the low latency characteristics of the shared memory channel can be used to exchange data directly in the memory without passing through the network stack, thereby greatly reducing the data synchronization delay and improving the throughput. Between different nodes, the adaptive data transmission channel can adjust the transmission strategy according to the real-time network conditions, such as dynamically adjusting the queue depth, transmission block size and timeout period to cope with network fluctuations and ensure the stability and efficiency of data synchronization. This method can solve the technical problem of how to apply NVMe-oF to distributed storage systems to improve data synchronization efficiency. Compared with the shortcomings of traditional synchronization methods with high latency within the same node and low efficiency across nodes, this application further reduces unnecessary network transmission, reduces the overall operating costs of the data center, and improves data synchronization efficiency.

[0034] Furthermore, in an exemplary embodiment, the process of creating a shared memory channel may include: determining whether the memory capacity of the physical server is greater than a preset capacity, wherein the preset capacity represents the capacity corresponding to the minimum memory requirement for creating the shared memory channel; if the memory capacity is greater than the preset capacity, obtaining a shared memory identifier for the shared memory channel, wherein the shared memory identifier is generated based on the node identifier of the same node and the storage pool identifier corresponding to the same node; and determining the shared memory channel based on the channel indicated by the shared memory identifier. It should be noted that the prerequisite for creating a shared memory channel is that the physical server has sufficient memory capacity to ensure that performance bottlenecks are not caused by insufficient memory during data synchronization. The preset capacity in this embodiment is a threshold and is also used to determine whether the server meets the conditions for creating a shared memory channel. Once the memory capacity meets the requirements, by generating a shared memory identifier, the shared memory area used for data synchronization can be uniquely identified, ensuring the security and accuracy of data transmission. The shared memory identifier combines the node identifier and the storage pool identifier, which enables precise location of data requiring synchronization even in complex environments with multiple nodes and storage pools. Therefore, this embodiment solves the problem of insufficient memory that may be encountered on a physical server with limited resources, and also ensures the correctness and reliability of data synchronization through the identification mechanism.

[0035] Optionally, for example, if it is detected that the client and the OSD (object storage device) belong to the same physical node (with the same node ID) and the node's memory capacity is ≥ 8GB (the preset capacity that meets the minimum shared memory requirements), the shared memory channel creation process is triggered. Specifically, the OSD generates a globally unique shared memory identifier (key = node ID + storage pool ID hash value) through a shmget function call to ensure isolation of shared memory channels between different storage pools within the same node. If creation fails (e.g., due to insufficient permissions), it automatically downgrades to an mmap anonymous mapping. The node ID corresponds to the node identifier, and the storage pool ID corresponds to the storage pool identifier. Ensuring isolation of shared memory channels between different storage pools within the same node can be understood as if the same node corresponds to different storage pools. The channels indicated by the shared memory identifier include multiple channels corresponding to different storage pool identifiers, and these multiple channels are isolated from each other. The shmget function allows applications to create or access existing shared memory segments. mmap is a system call for memory-mapped files. It not only supports mapping files into memory, but can also be used to create anonymous mappings—virtual memory areas not associated with any files—through which multiple processes can communicate.

[0036] Furthermore, in an exemplary embodiment, the network performance parameters include at least one of the following: queue depth, transmission block size, and timeout period. The adaptive data transmission channel is configured based on the network performance parameters of the physical server, including at least one of the following schemes: determining the transmission delay and queue fill rate of the physical server, determining a first adjustment strategy for the queue depth based on the transmission delay and queue fill rate, and reconfiguring the queue depth using the first adjustment strategy; determining the bandwidth utilization and data type of the physical server, determining a second adjustment strategy for the transmission block size based on the bandwidth utilization and data type, and reconfiguring the transmission block size using the second adjustment strategy; and, if it is determined that the timeout period is greater than a preset time, determining a packet retransmission strategy based on the packet loss type, and reconfiguring the retransmitted packets of the adaptive data transmission channel according to the packet retransmission strategy. The design of the adaptive data transmission channel aims to optimize cross-node data transmission efficiency. This embodiment maximizes the utilization of the physical server's network resources while reducing data transmission delay and packet loss rate by dynamically adjusting parameters such as queue depth, transmission block size, and timeout period. For example, when an increase in transmission delay and a high queue fill rate are detected, the adjustment strategy will reduce the queue depth to avoid increased delay caused by excessive packet accumulation. Similarly, adjusting the transmission block size based on bandwidth utilization and data type can reduce the possibility of network congestion while ensuring data transmission rate.

[0037] Optionally, for example, for queue depth adjustment, the default value of the initial queue depth is adjusted based on the latency and queue fill rate, such as the default value of 128. If the latency is less than 50μs and the queue fill rate is less than 50%, increase it by 32 each time (up to a maximum of 1024) to improve concurrency. If the latency is greater than 200μs or the queue fill rate is greater than 80%, reduce it by 16 each time (minimum of 16) to avoid request accumulation timeouts. For random read and write workloads (such as AI training data), the queue depth is fixed at 64 to reduce head blocking. For sequential read and write workloads (such as video streaming), the queue depth can be dynamically increased to 256.

[0038] The transmission block size is adjusted based on bandwidth and data type. For example, the basic block size is 4KB. When bandwidth utilization is less than 30%, indicating network idleness, the block size is increased to 16KB-64KB. When bandwidth utilization is greater than 70%, indicating network congestion, the block size is reduced to 1KB-2KB.

[0039] For erasure coded stripes (such as 8+4 configuration), the block size is forced to align with the stripe unit. For example, if the stripe size is 256KB, the block size is set to 32KB, and 8 data blocks just fill the stripe.

[0040] For timeouts and retransmissions, the timeout is calculated based on "average delay + 3 standard deviations" and dynamically refreshed every 1 second. In the case of random packet loss, only the lost packets are retransmitted, with a retransmission interval of half the timeout period to prevent increased network congestion. In the case of batch packet loss, new requests are suspended, and after a 50ms wait, the connection is reestablished and all unacknowledged data is retransmitted.

[0041] Furthermore, in an exemplary embodiment, if both the shared memory channel and the adaptive data transmission channel have been successfully created, at least the cache hit rate and access latency of the shared memory channel, as well as the bandwidth utilization of the adaptive data transmission channel, may be obtained. If the cache hit rate of the shared memory channel is lower than a first threshold for multiple consecutive times and the access latency is greater than the first threshold, the shared memory channel is switched to the adaptive data transmission channel. If the bandwidth utilization is lower than a second threshold and the remaining resources of the shared memory channel are greater than the second threshold, the adaptive data transmission channel is switched to the shared memory channel. This embodiment introduces a flexible switching mechanism that enables switching between the shared memory channel and the adaptive data transmission channel based on real-time performance indicators to maintain optimal data synchronization performance. For example, when the cache hit rate and access latency of the shared memory channel no longer meet the requirements for efficient synchronization, the shared memory channel automatically switches to the adaptive data transmission channel, leveraging its adaptability to network conditions to ensure data synchronization stability. Conversely, if the bandwidth utilization of the adaptive data transmission channel is too low and the shared memory channel still has sufficient resources, the shared memory channel is switched back to fully utilize local resources and improve data synchronization efficiency. This mechanism solves the performance bottleneck problem that may exist in a single channel in all scenarios.

[0042] Furthermore, in an exemplary embodiment, the following technical solution is proposed: when both the shared memory channel and the adaptive data transmission channel have been successfully created, the current status of the shared memory channel and the current status of the adaptive data transmission channel are obtained; when the current status of the shared memory channel indicates that the shared memory channel has experienced a memory leak or access conflict, the shared memory channel is switched to the adaptive data transmission channel; and when the current status of the adaptive data transmission channel indicates that the adaptive data transmission channel has experienced a batch packet loss or link interruption, the adaptive data transmission channel is switched to the shared memory channel. This embodiment continuously monitors the operating status of the shared memory channel and the adaptive data transmission channel, such as events such as memory leaks, access conflicts, batch packet loss, and link interruptions, to promptly detect and address potential performance or stability issues. When a memory leak or access conflict occurs in the shared memory channel, the adaptive data transmission channel is immediately switched to, avoiding waste of memory resources and data synchronization errors. When the adaptive data transmission channel experiences batch packet loss or link interruption, the shared memory channel is switched back to, leveraging its low latency to quickly restore data synchronization. This not only improves the reliability of data synchronization but also increases the efficiency of data synchronization by avoiding prolonged channel failures.

[0043] Furthermore, in an exemplary embodiment, a cache hit rate, a delay compliance rate, and a memory usage rate monitored in real time may also be obtained; a first availability score of the shared memory channel may be determined based on a first set of weight coefficients and a weighted sum of the cache hit rate, the delay compliance rate, and the memory usage rate, wherein the first set of weight coefficients includes a first weight corresponding to the cache hit rate, a first weight corresponding to the delay compliance rate, and a first weight corresponding to the memory usage rate; a second availability score of the adaptive data transmission channel may be determined based on a second set of weight coefficients and a weighted sum of the cache hit rate, the delay compliance rate, and the memory usage rate, wherein the second set of weight coefficients includes a second weight corresponding to the cache hit rate, a second weight corresponding to the delay compliance rate, and a second weight corresponding to the memory usage rate; if it is determined that the difference between the first availability score and the second availability score is greater than a preset difference and persists for at least two monitoring cycles, a target channel may be determined from the shared memory channel and the adaptive data transmission channel, wherein the target channel is the channel having the larger value between the first availability score and the second availability score; and data synchronization between the client and the object storage device may be performed via the target channel. This embodiment introduces an availability scoring mechanism to intelligently evaluate the real-time performance of shared memory channels and adaptive data transmission channels, thereby selecting the optimal data synchronization path. The scoring mechanism takes into account multiple key indicators such as cache hit rate, latency compliance rate, and memory utilization, and uses different weight coefficients to reflect the impact of each indicator on data synchronization performance. For example, for scenarios that are highly dependent on cache, a higher weight is set for cache hit rate to solve the problem of blind channel selection. The scoring criteria can also be dynamically adjusted according to the characteristics of the specific application scenario to ensure that the data synchronization strategy always meets business needs.

[0044] Furthermore, in an exemplary embodiment, after both the shared memory channel and the adaptive data transmission channel have been successfully created, for the case where the client and the object storage device are located in the same node, data synchronization between the client and the object storage device is performed through the shared memory channel; when it is determined that the real-time availability score of the shared memory channel is lower than the preset score, the shared memory channel is switched to the adaptive data transmission channel; for the case where the client and the object storage device are located in different nodes, data synchronization between the client and the object storage device is preferentially performed through the adaptive data transmission channel; when it is detected that the client and the object storage device are temporarily deployed in the same node, the adaptive data transmission channel is switched to the shared memory channel. This embodiment significantly improves the efficiency of data redundancy protection by optimizing the erasure code calculation process. Before performing the erasure code calculation, check whether there is already a calculation result in the shared memory cache. If there is a hit, the result in the cache is directly used, avoiding the waste of resources caused by repeated calculations and improving calculation efficiency. In addition, in the event of a miss, the calculation results will be stored in the cache to provide a fast response for subsequent calculation requests, further reducing calculation delays, optimizing the performance of data redundancy protection, and reducing the operating costs of the data center by reducing the consumption of computing resources.

[0045] Furthermore, in an exemplary embodiment, the process of configuring an adaptive data transmission channel according to the network performance parameters of the physical server includes: determining a queue depth parameter of the adaptive data transmission channel according to the following formula:

[0046] ;

[0047] Wherein, Optimal_Queue_Depth is the queue depth parameter, BDP is the bandwidth-delay product of the physical server, Block_Size is the size of the data block to be sent by the object storage device, MAX_QUEUE_DEPTH is the maximum queue depth, α is the adjustment coefficient, and Workload_Factor is the workload factor. The adaptive data transmission channel is configured based on the queue depth parameters. By adaptively adjusting the queue depth parameters, cross-node data transmission performance is optimized. The queue depth parameter is determined by comprehensively considering the network delay product (BDP), data block size (Block_Size), maximum queue depth (MAX_QUEUE_DEPTH), and workload factor (Workload_Factor), ensuring that the data transmission channel operates optimally under different network conditions and loads. For example, when network latency is low and data blocks are large, the queue depth can be set relatively high to fully utilize network bandwidth and improve transmission efficiency.

[0048] Furthermore, in an exemplary embodiment, before any object storage device in the physical server performs erasure code calculation, it can detect whether the calculation result of the erasure code calculation hits in the shared memory cache of the physical server; if the calculation result hits in the shared memory cache, the calculation result is directly obtained from the shared memory cache; if the calculation result does not hit in the shared memory cache, the object storage device is instructed to perform the erasure code calculation and store the obtained calculation result in the shared memory cache. This embodiment introduces a hit rate detection mechanism to further improve the utilization efficiency of the shared memory cache. By monitoring the hit rate of the shared memory cache in real time, the effectiveness of the cache can be accurately assessed, and then a decision can be made as to whether to directly obtain the calculation result from the cache or perform a new calculation and store the result in the cache. This can significantly reduce computational latency and improve overall computational efficiency. At the same time, by continuously optimizing the cache strategy, such as eliminating infrequently accessed data, it can ensure that cache space is effectively utilized, avoid resource waste, optimize data redundancy protection performance, and reduce the operating costs of the data center by reducing computing resource consumption.

[0049] By way of example, a method for achieving replica consistency of an erasure code pool over a TCP / IP network, i.e., data synchronization, is provided, including the following steps:

[0050] 1. Data writing and encoding: The client sends data to the OSD. The OSD divides the data into data blocks according to the erasure code configuration (such as 4+2) and generates parity blocks through encoding.

[0051] 2. Stripe transmission and storage: OSD sends data blocks and check blocks to corresponding slave OSDs through TCP / IP network, and each OSD stores the data persistently.

[0052] 3. Confirmation and consistency marking: After receiving data from OSD, confirmation is returned to OSD. After OSD collects all confirmations, it marks the data as "consistent" and responds to the client.

[0053] 4. Fault detection and repair: When the Monitor detects an OSD failure or data inconsistency, it triggers the repair process: reading valid blocks from other OSDs via TCP / IP, using erasure codes to reconstruct the lost data, and writing it to the new OSD.

[0054] 5. Regular consistency check: The background thread regularly scans the erasure code stripes and detects data errors through checksum comparison. If inconsistencies are found, they are repaired through TCP / IP synchronization.

[0055] Furthermore, in an exemplary embodiment, the scheme for detecting whether the calculation result of the erasure code calculation hits the shared memory cache of the physical server includes: determining the hit rate of the shared memory cache at the current moment according to the following formula.

[0056] ;

[0057] Where Cache_Hit_probability is the hit rate of the shared memory cache at the current moment, λ is the time decay coefficient, μ is the access frequency coefficient, Δt is the time interval between the last access to the shared memory cache and the current moment, and N is the number of accesses to the shared memory cache within the historical time period. If the hit rate is greater than a third threshold, the calculation result is determined to have hit the shared memory cache; if the hit rate is less than the third threshold, the calculation result is determined to have missed the shared memory cache. This embodiment implements intelligent management of the shared memory cache by introducing a hit rate detection formula. This formula comprehensively considers the time decay coefficient (λ), the access frequency coefficient (μ), the time interval (Δt), and the number of historical accesses (N). It accurately predicts the cache hit probability at the current moment and then determines whether to read data from the cache. This solves the "hot and cold" data problem in cache management, automatically identifying and prioritizing frequently accessed "hot" data while eliminating "cold" data that has not been accessed for a long time, ensuring efficient use of cache space.

[0058] Furthermore, in an exemplary embodiment, the time interval between the last time the shared memory cache was accessed and the current moment, as well as the number of accesses to the shared memory cache within a historical time period, can also be obtained; when the time interval is greater than a fourth threshold value and the number of accesses is less than the fourth threshold value, the shared memory cache is eliminated, and a new shared memory cache is determined from the physical server based on an election algorithm. This embodiment designs a dynamic adjustment mechanism to improve the utilization of the shared memory cache. Specifically, by monitoring the time interval and the number of historical accesses, caches that have not been accessed for a long time and have a very low access frequency can be identified and eliminated, freeing up valuable memory resources. At the same time, the shared memory cache is re-determined based on the election algorithm, ensuring the fairness of cache distribution and the balance of data access. This method solves the problem of unbalanced cache resource allocation, can effectively avoid excessive occupation of memory resources of some nodes, and improve resource utilization and data synchronization efficiency of the entire system.

[0059] Furthermore, in an exemplary embodiment, it is also possible to detect whether there is a difference in the data versions of any two object storage devices in the physical server based on the vector clock; if there is a difference in the data versions of any two object storage devices in the physical server, determine the difference value of the data version difference; if the difference value is less than a fifth threshold, determine the incremental data between the data versions of the any two object storage devices, wherein the incremental data is the difference data of the first object storage device relative to the second object storage device, and the any two object storage devices include: the first object storage device and the second object storage device; send the incremental data to the second object storage device to make the data versions of the first object storage device and the second object storage device consistent. This embodiment solves the problem of inconsistent data versions in a distributed storage system by introducing a vector clock and an incremental synchronization mechanism. Specifically, by detecting data version differences through a vector clock, it is possible to accurately determine whether the data between any two object storage devices needs to be synchronized, as well as the scope and magnitude of the synchronization. When the difference value is small, only incremental data is transmitted instead of full data, which significantly reduces the amount of data transmitted and reduces the network load.

[0060] Furthermore, in an exemplary embodiment, the process of determining the incremental data between the data versions of any two object storage devices includes: determining the incremental data using the following formula:

[0061] ;

[0062] Where Sync_Data_Size represents the incremental data, Base_Size represents the base data block size, β represents the version difference coefficient, Version_Diff represents the number of version differences, γ represents the compression ratio determined based on data characteristics, and Compression_Ratio represents the compression ratio. This embodiment achieves efficient data synchronization transmission by meticulously calculating the incremental data volume. The incremental data determination formula comprehensively considers the base data block size (Base_Size), the version difference coefficient (β), the number of version differences (Version_Diff), and the data compression ratio (γ). This accurately calculates the incremental data volume required for transmission, avoiding repeated transmission of the full data. For example, when the number of version differences is small, the β coefficient decreases, thereby reducing the incremental data size and alleviating the network transmission burden. Furthermore, data compression can reduce the amount of transmitted data by increasing the compression ratio (γ), even in the case of large version differences. This approach not only optimizes data synchronization transmission efficiency but also reduces network traffic, lowering data center operating costs.

[0063] Furthermore, in an exemplary embodiment, upon detecting that at least two object storage devices in the physical server have failed, a repair priority for the at least two object storage devices can be determined; and a repair task can be performed on a third object storage device with the highest repair priority. This embodiment optimizes the repair process for failed object storage devices by intelligently determining repair priorities. When multiple object storage device failures are detected, the repair priority of each device is comprehensively evaluated based on a series of factors, such as data importance, failure risk, and data inconsistency time. This approach ensures that critical data and high-risk devices are repaired first, improving the stability and security of the data synchronization process.

[0064] Furthermore, in an exemplary embodiment, the process of determining the repair priorities of the at least two object storage devices includes: determining the repair priorities of the at least two object storage devices by using the following formula:

[0065] ;

[0066] Here, repair_priority is the repair priority, importance is the data importance factor of the corresponding object storage device, failure_risk is the failure risk factor of the corresponding object storage device, age is the data inconsistency duration of the corresponding object storage device, and decay_rate is the decay rate. This embodiment implements intelligent repair sorting for faulty object storage devices by introducing a repair priority calculation formula. This formula comprehensively considers data importance (importance), failure risk (failure_risk), data inconsistency duration (age), and decay rate (decay_rate), and can accurately assess the repair urgency of each storage device. For example, the more important the data, the higher the failure risk, and the longer the data inconsistency duration, the higher its repair priority. This method solves the resource allocation problem in fault repair, ensures that critical data and high-risk devices can be repaired in a timely manner, and improves data security.

[0067] Furthermore, in an exemplary embodiment, the scheme for executing the repair task on the third object storage device with the highest repair priority includes: obtaining the data blocks required for repairing the third object storage device from the replica pool or erasure code pool of the physical server; when the third object storage device receives the required data blocks, verifying the required data blocks, and confirming that the repair task for the third object storage device has been completed after the verification passes. This embodiment improves the fault recovery efficiency of the object storage device by optimizing the fault repair process. When repairing the highest priority device fault, the required repair data blocks are quickly obtained from the replica pool or erasure code pool, avoiding redundant transmission of the full amount of data and saving network resources. At the same time, through data block verification, the integrity and accuracy of the repaired data are ensured, and secondary failures caused by data errors are avoided. This method not only optimizes the speed and accuracy of fault repair, but also improves user experience. At the same time, through priority repair and data verification, data security can be enhanced and overall system performance can be improved.

[0068] In one embodiment, Figure 2 As shown, a distributed storage system architecture on which the execution of a data synchronization method depends is provided. First, the distributed storage system architecture is described here.

[0069] Distributed storage systems utilize a decentralized, distributed architecture. Core components include clients, a control plane (Monitor / MDS), metadata servers (MDS), and object storage devices (OSDs). These modules collaborate over high-speed networks (such as Ethernet and InfiniBand) to store and manage data. The following describes the architecture's layers. "Monitor" in Chinese means "monitor."

[0070] Within the storage pool, the "controllable, scalable, distributed data balancing algorithm" can be understood as the CRUSH algorithm, which determines the distribution of data based on the hardware hierarchy (such as racks, servers, hard drives, etc.), thereby achieving balanced data distribution.

[0071] The client layer is the entry point for users to interact with distributed storage, providing block, object, and file interfaces. Core components include client libraries (such as the librados library), which encapsulate the underlying Application Programming Interface (API) and interact with the cluster. It also includes client modules for algorithms (such as the CRUSH algorithm) that locally cache CRUSH maps and calculate data storage locations. The CRUSH map is an algorithm used in the Ceph distributed file system to determine data location. It considers information such as node relationships, node weights, and failure domains to ensure more even and controllable data distribution. For example, if you enter "object name, storage pool rules (number of replicas, failure domain hierarchy)," the object name is hashed to generate a random number. Based on the storage pool rules, the CRUSH bucket hierarchy (e.g., host -> cabinet -> data center) is traversed, and OSDs are selected based on weights to ensure balanced data distribution and meet redundancy policies (such as cross-cabinet replica deployment).

[0072] The control plane maintains or extends cluster metadata, such as CRUSH maps and authentication information, as well as node locations and channel capabilities to ensure multi-node consistency. It pulls the latest cluster map upon initial client access and subsequently synchronizes updates via heartbeats. It manages file system metadata (directory trees, permissions) and caches hotspot metadata to accelerate access.

[0073] The data plane (OSD cluster) consists of OSD nodes, a shared memory pool, adapters (such as NVMe-oF adapters), and an erasure code calculation engine. OSD nodes include CPUs (central processing units), memory, disks, and network interfaces (Gigabit / 10Gbit network cards). The data plane is responsible for data storage, replication, erasure coding, consistency maintenance, and cluster topology awareness through the OSD map.

[0074] This application combines the following multiple functional modules to collaboratively implement a solution for data consistency synchronization from the perspective of functional implementation. These functional modules are as follows.

[0075] The location awareness module implements a multi-level location detection algorithm to distinguish between same-node, same-rack, and cross-node scenarios. The location awareness module detects the location relationship between the client and OSD (same-node and cross-node), dynamically selects a shared memory channel or an adaptive NVMe-oF channel (corresponding to an adaptive data transmission channel), and uses the performance monitoring module to evaluate channel performance in real time and trigger dynamic switching.

[0076] The adaptive NVMe-oF channel management module maintains two transmission paths: shared memory channels and adaptive NVMe-oF channels. It dynamically selects transmission channels based on node location information, using shared memory channels within the same node to avoid network protocol stack overhead and adaptive NVMe-oF channels across nodes, adjusting parameters based on network status. Specifically, it includes a shared memory channel controller, an NVMe-oF parameter optimizer (queue depth / block size), and a channel switching decider (based on the Channel_Score algorithm).

[0077] The adaptive NVMe-oF channel selection process is as follows.

[0078] 1. The location awareness module detects the location relationship between the client and the OSD.

[0079] 2. If they are on the same node, create a shared memory channel.

[0080] 3. If it is across nodes, detect network bandwidth and latency and configure adaptive NVMe-oF channels.

[0081] 4. The performance monitoring module evaluates channel performance in real time and triggers channel switching when necessary.

[0082] Among them, the channel score Channel_Score needs to be calculated during adaptive NVMe-oF channel selection. The formula is as follows.

[0083] .

[0084] ω1, ω2, ω3, and ω4 are all weight coefficients, and their default values ​​are, for example, [0.4, 0.3, 0.2, 0.1]. Locality_Factor represents the location factor, with same-node = 1.0, same-rack = 0.7, and cross-rack = 0.3.

[0085] Bandwidth_Factor represents the bandwidth factor, calculated as min(actual_bandwidth, ideal_bandwidth) / ideal_bandwidth. Latency_Factor represents the latency factor, calculated as 1 / (1+latency). Workload_Factor represents the workload factor, with random read / write = 0.8 and sequential read / write = 1.0.

[0086] The algorithm formula of the NVMe-oF queue depth optimization algorithm is as follows.

[0087] .

[0088] Optimal_Queue_Depth is the queue depth parameter, BDP (bandwidth-delay product) = network bandwidth × RTT, Block_Size is the data block size, MAX_QUEUE_DEPTH is the maximum queue depth, the default value is 1024, α is the adjustment coefficient, the default value is 0.2, Workload_Factor is the workload factor, random read and write = 0.8, sequential read and write = 1.2.

[0089] The erasure code optimization module is used to implement a shared memory-based erasure code calculation cache mechanism, optimize encoding matrix operations, and improve computational efficiency. Specifically, before the primary OSD performs erasure code calculations, it first queries the shared memory cache. If a hit is found, the calculation result is reused. If a hit is found, the calculation is performed and cached. Old cache is eliminated through time and access frequency strategies, reducing duplicate calculations within the same node. Erasure code calculation optimization driven by shared memory reduces duplicate calculations within the same node, lowers CPU usage, improves system availability, improves hardware resource utilization, reduces dependence on high-end network hardware, and reduces the consumption of storage and computing resources.

[0090] The process of erasure code calculation optimization is as follows.

[0091] 1. Before the primary OSD performs erasure code calculation, it checks the shared memory cache.

[0092] 2. If the cache hits, use the cached result directly.

[0093] 3. If the cache misses, perform the calculation and store the result in shared memory.

[0094] 4. Cache elimination strategy based on time and access frequency.

[0095] Among them, the algorithm formula of the shared memory cache hit rate prediction algorithm is as follows.

[0096] .

[0097] Cache_Hit_probability is the hit rate of the shared memory cache at the current moment, λ is the time decay coefficient, which defaults to 0.1, μ is the access frequency coefficient, which defaults to 0.2, △t is the interval (in seconds) from the last access time to the present, and N is the number of accesses in the past time period.

[0098] The algorithm formula of the erasure code repair priority algorithm is as follows.

[0099] .

[0100] repair_priority is the repair priority, importance is the data importance factor, for example, critical data = 2, common data = 1. failure_risk is the failure risk factor, calculated based on the health status of the node / disk. age is the time (in hours) that the data is inconsistent, and decay_rate is the decay rate, with the default value being 0.5.

[0101] The consistency assurance module is used to design a vector clock-based version control mechanism to implement an efficient incremental synchronization protocol. Specifically, it uses vector clocks to detect data version differences. When a threshold is exceeded, a full synchronization is performed. Otherwise, only the changed data blocks are transmitted (incremental synchronization). This is accomplished through optimized channels (shared memory or NVMe-oF). The incremental synchronization protocol transmits only changed data blocks, reducing the amount of synchronized data and significantly reducing network traffic. This enhances data consistency assurance, shortens fault repair time, and allows for rapid recovery of critical data.

[0102] The data synchronization process is as follows.

[0103] 1. Detect data version differences based on vector clocks.

[0104] 2. If the difference exceeds the threshold, perform a full synchronization.

[0105] 3. Otherwise, perform incremental synchronization and transfer only the changed data blocks.

[0106] 4. Use optimized channels to complete data transmission.

[0107] The fault repair process is as follows.

[0108] 1. After detecting an OSD failure, calculate the repair priority.

[0109] 2. Select the optimal repair path and channel.

[0110] 3. Use erasure codes to reconstruct lost data.

[0111] 4. Update the cluster status to confirm that the repair is complete.

[0112] The calculation formula for the incremental synchronization data volume is as follows.

[0113] .

[0114] Sync_Data_Size represents the incremental data, Base_Size represents the size of the basic data block, β represents the version difference coefficient, the default value is 0.3, Version_Diff represents the number of version differences, γ represents the compression ratio, which is calculated based on data characteristics, and Compression_Ratio is the compression ratio.

[0115] The performance monitoring module collects channel performance metrics (latency, bandwidth, and CPU utilization) in real time and predicts the optimal channel selection strategy based on a reinforcement learning model. Upon detecting an OSD failure, it calculates the repair priority based on parameters such as data importance and failure risk, selects the optimal path and channel, reconstructs the data using erasure coding, and updates the cluster status.

[0116] Based on the above content, this application can dynamically select shared memory channels or adaptive NVMe-oF channels through the location awareness module, use shared memory to realize cache reuse of erasure code calculation results, combine vector clocks to achieve incremental synchronization, and optimize queue depth, repair priority and other parameters through intelligent algorithms. It can dynamically adjust strategies according to node location, network status and workload, solve problems such as high network overhead, repeated calculations, and low synchronization efficiency in traditional solutions, reduce synchronization delays on the same node, improve bandwidth utilization across nodes, and significantly improve the performance, reliability and resource utilization of distributed storage systems.

[0117] Furthermore, the data synchronization process is described in conjunction with the following embodiments.

[0118] Example 1: Same-node erasure code calculation optimization.

[0119] Scenario Description: The client and three OSDs reside on the same physical server (i.e., the same node), using a "4+2" erasure coding configuration to write video streams. The client program and the three OSD processes responsible for storing data run on the same physical server. Video stream data is written using an erasure coding configuration that divides data into four data blocks and two parity blocks. This scenario utilizes the server's internal shared memory to optimize data transmission and improve storage performance. The technical implementation steps are as follows.

[0120] 1. The location awareness module detects that all OSDs are on the same node as the client.

[0121] 2. Create a shared memory channel and allocate a 2GB shared memory pool.

[0122] 3. Before the primary OSD performs erasure code calculation, it checks the shared memory cache.

[0123] 4. Cache miss, perform the calculation and store the result in shared memory.

[0124] 5. Other OSDs read the calculation results directly from the shared memory to avoid repeated calculations.

[0125] 6. Data synchronization is completed through shared memory channels without the need for network transmission.

[0126] In same-node scenarios, zero-copy transmission is achieved through shared memory channels, eliminating TCP / IP protocol stack overhead, reducing data synchronization latency, and improving throughput. For example, in the same-node erasure code calculation optimization scenario, the erasure code calculation latency for video stream writes is reduced from 1.8ms in the traditional solution to 0.5ms, and the synchronization bandwidth is increased from 2.5GB / s to 6.8GB / s. In terms of optimizing computing resource utilization, the shared memory erasure code calculation cache mechanism avoids repeated calculations within the same node, reducing CPU utilization. For example, in a (4+2) erasure code configuration, multiple OSDs within the same node reuse calculation results, reducing encoding calculation overhead.

[0127] Example 2: Cross-node NVMe-oF channel optimization.

[0128] Scenario description: The client and three OSDs are distributed across different racks, with a 100Gbps network bandwidth. (8+4) erasure coding is used to process big data analytics workloads. The technical implementation steps are as follows.

[0129] 1. The location awareness module detects that all OSDs and clients are across nodes.

[0130] 2. Detect network bandwidth and latency and configure adaptive NVMe-oF channels.

[0131] 3. Calculate the optimal queue depth according to the above algorithm formula.

[0132] 4. Configure the intelligent I / O scheduling policy.

[0133] 5. Use multipath I / O to improve reliability.

[0134] 6. Data is transmitted in blocks, and each block is sent through an independent NVMe-oF connection.

[0135] In cross-node scenarios, adaptive NVMe-oF channels dynamically adjust parameters (such as queue depth and block size) based on network status, improving network bandwidth utilization. For example, when processing big data analytics workloads on a 100Gbps network, cross-rack transmission latency can be reduced from 220μs to 150μs, shortening job completion time by 35%.

[0136] Example 3: Application of incremental synchronization protocol.

[0137] Scenario description: There is a data version discrepancy between two OSDs, and synchronization is required. The technical implementation steps are as follows.

[0138] 1. Based on the vector clock detection version difference, there are 3 versions.

[0139] 2. Calculate the amount of incremental synchronization data.

[0140] 3. Only the changed data is transmitted, not the full data.

[0141] 4. Use adaptive NVMe-oF channels to transmit incremental data.

[0142] 5. The receiver applies the incremental data and updates the version.

[0143] Because the incremental synchronization protocol only transmits changed data blocks, it reduces the amount of synchronization data and network traffic. When there are three version differences between two OSDs, incremental synchronization only requires transferring 4.56MB of data, reducing data transmission by approximately 70% compared to full synchronization. Vector clock-based version control and the incremental synchronization protocol ensure real-time detection of data version differences and accurate synchronization. This reduces fault repair time and significantly shortens the window of data inconsistency.

[0144] In order to achieve the organic combination of the algorithms and processes described above in this application, this application embeds each core algorithm into the key process nodes of the distributed storage system through a closed-loop data interaction between modules and a real-time transmission mechanism of algorithm parameters, as follows.

[0145] In one embodiment, an adaptive channel selection algorithm and location awareness process are proposed: location awareness → channel configuration → performance monitoring → dynamic switching.

[0146] During the location awareness phase, the location awareness module obtains node topology information by extending CRUSHMap, calculates the Locality_Factor (same node = 1.0), and triggers the shared memory channel creation process (allocating a 2GB memory pool, as in step 2 of Example 1 above). The Channel_Score algorithm is then called. If the Locality_Factor weight (w1 = 0.4) dominates the score, the shared memory channel is directly selected.

[0147] During the channel configuration phase, when Locality_Factor = 0.3 (across racks), the NVMe-oF queue depth is calculated using the Optimal_Queue_Depth algorithm (e.g., BDP = 100 Gbps × 50 μs = 5 MB, Block_Size = 512 KB, the calculated queue depth = min(5 MB / 512 KB, 1024) × (1 + 0.2 × 1.2) = 9 × 1.24 ≈ 11, corresponding to step 3 of Example 2).

[0148] In the performance monitoring → dynamic switching stage, the performance monitoring module calculates Cache_Hit_Probability in real time. If the shared memory channel hit rate is lower than the threshold (such as less than 30%), the Channel_Score algorithm is re-called and switched to the NVMe-oF channel (corresponding to step 4 of Example 3).

[0149] The adaptive channel selection mechanism dynamically adapts to different topology scenarios, such as within the same node, within the same rack, and across racks, improving adaptability to heterogeneous network environments such as 10G / 25G / 100G. For example, in cross-node scenarios, the NVMe-oF queue depth can be dynamically adjusted based on the BDP to ensure optimal performance under different bandwidth conditions.

[0150] In one embodiment, an erasure code calculation optimization algorithm and data processing flow are proposed: cache query before calculation → encoding execution → result cache → elimination.

[0151] During the cache query phase, before the primary OSD performs erasure code calculation, it uses the Cache_Hit_Probability algorithm to predict the hit rate (e.g., Δt = 10s, N = 5, the calculated hit rate is ≈ 0.22, step 3 of Example 1). If a hit is found, the cache result is directly reused.

[0152] In the encoding execution → result cache stage, if the cache misses, the erasure code calculation is performed and the Sync_Data_Size algorithm is called to calculate the cache data size (Base_Size = 4MB, Version_Diff = 0, compression ratio = 0.4, 4MB × 1 × 0.6 = 2.4MB, step 4 of Example 1), and stored in the shared memory.

[0153] In the elimination phase, based on the time decay coefficient λ=0.1 and the access frequency μ=0.2, caches that have not been accessed for a long time (eg, greater than 60s and equal to 0) are eliminated to maintain cache efficiency (corresponding to step 6 of Example 1).

[0154] Erasure code calculation cache and incremental synchronization reduce redundant data transmission and calculation, improve storage resource utilization, reduce computing resource consumption, and thus reduce the overall data center TCO (total cost of ownership).

[0155] In one embodiment, a connection process between a consistency synchronization algorithm and a data transmission process is proposed: version detection → synchronization strategy decision → channel transmission → verification.

[0156] During the version detection → synchronization strategy decision phase, the consistency assurance module calculates Version_Diff using the vector clock. If the number of different versions is greater than 2 (corresponding to step 1 in Example 3), a full synchronization is triggered. Otherwise, the Sync_Data_Size algorithm is used to calculate the incremental data size (Base_Size = 4MB, β = 0.3, Version_Diff = 3, γ = 0.4, resulting in 4MB × (1 + 0.9) × 0.6 = 4.56MB, corresponding to step 2 in Example 3).

[0157] In the channel transmission → verification phase, the incremental data is configured with the NVMe-oF channel (queue depth = 11) through the Optimal_Queue_Depth algorithm, and the block size is dynamically adjusted in combination with the Bandwidth_Factor (e.g., actual bandwidth 80 Gbps, ideal bandwidth 100 Gbps, block size = 512 KB × 0.8 = 409.6 KB, corresponding to step 4 of Example 3).

[0158] In one embodiment, a collaborative process between a fault repair algorithm and a cluster management process is proposed: fault detection → priority calculation → path selection → reconstruction.

[0159] In the fault detection → priority calculation phase, when an OSD fails, the priority is calculated using the Repair_Priority algorithm (Importance = 2, Failure_risk = 0.8, Age = 2h, Decay_rate = 0.5, resulting in (2 × 0.8) / (1 + 2 × 0.5) = 0.8, Example 2, Step 1), and high-priority data is repaired first.

[0160] In the path selection → reconstruction phase, select the same-node shared memory channel (if available) or optimize the NVMe-oF channel (cross-node), and use the Channel_Score algorithm to evaluate the channel score (e.g., Locality_Factor = 0.7, Bandwidth_Factor = 0.9, the score is 0.4 × 0.7 + 0.3 × 0.9 = 0.55, corresponding to step 2 of Example 2) to ensure reconstruction efficiency.

[0161] This prioritization algorithm, based on data importance, failure risk, and repair cost, enables intelligent scheduling and rapid reconstruction of faulty data. This prioritizes the repair of critical data, ensuring that high-value data is restored first, improving system availability.

[0162] Based on the above solution, this application improves the efficiency of distributed storage consistency synchronization through technologies such as adaptive NVMe-oF channel management and shared memory computing cache. This technical approach based on real-time state perception and dynamic resource scheduling can be applied to heterogeneous storage resource management in edge computing scenarios. For example, a lightweight NVMe-oF gateway is deployed at the edge node, and combined with location perception and load prediction algorithms, the local NVMe storage and cloud storage resources of the edge device are dynamically scheduled to achieve efficient synchronization and cache updates of edge data. It can also be expanded to 5G network slicing scenarios. According to the QoS requirements of different slices, the NVMe-oF channel parameters are adaptively adjusted to ensure data consistency and low-latency transmission of high-priority services, providing more flexible storage solutions for scenarios such as edge intelligence and industrial Internet of Things.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0164] The embodiment of the present application also provides a data synchronization device. Figure 3 As shown, including:

[0165] A detection module 32 is used to detect the position relationship between the client and the object storage device;

[0166] A first synchronization module 34 is configured to create a shared memory channel and synchronize data between the client and the object storage device through the shared memory channel when the location relationship indicates that the client and the object storage device are located on the same node;

[0167] The second synchronization module 36 is used to configure an adaptive data transmission channel according to the network performance parameters of the physical server when the position relationship indicates that the client and the object storage device are located at different nodes, and synchronize data between the client and the object storage device through the adaptive data transmission channel, wherein the client and the object storage device are both located on the physical server.

[0168] Through this application, the positional relationship between the client and the object storage device is intelligently detected, and the optimal data synchronization path is dynamically selected, which significantly improves the data synchronization efficiency. For example, when the client and the object storage device are in the same physical server node, the low latency characteristics of the shared memory channel can be used to exchange data directly in the memory without passing through the network stack, thereby greatly reducing the data synchronization delay and improving the throughput. Between different nodes, the adaptive data transmission channel can adjust the transmission strategy according to the real-time network conditions, such as dynamically adjusting the queue depth, transmission block size and timeout period to cope with network fluctuations and ensure the stability and efficiency of data synchronization. This method can solve the technical problem of how to apply NVMe-oF to distributed storage systems to improve data synchronization efficiency. Compared with the shortcomings of traditional synchronization methods with high latency within the same node and low efficiency across nodes, this application further reduces unnecessary network transmission, reduces the overall operating costs of the data center, and improves data synchronization efficiency.

[0169] Furthermore, in an exemplary embodiment, the first synchronization module is also used to: determine whether the memory capacity of the physical server is greater than a preset capacity, the preset capacity representing the capacity corresponding to the minimum memory requirement for creating the shared memory channel; when the memory capacity is greater than the preset capacity, obtain the shared memory identifier of the shared memory channel, wherein the shared memory identifier is generated based on the node identifier of the same node and the storage pool identifier corresponding to the same node; and determine the shared memory channel according to the channel indicated by the shared memory identifier.

[0170] Further, in an exemplary embodiment, the network performance parameters include at least one of the following: queue depth, transmission block size, timeout period, and a second synchronization module, and are also used to implement at least one of the following: determining the transmission delay and queue filling rate of the physical server, determining a first adjustment strategy for the queue depth based on the transmission delay and the queue filling rate, and reconfiguring the queue depth using the first adjustment strategy; determining the bandwidth utilization and data type of the physical server, determining a second adjustment strategy for the transmission block size based on the bandwidth utilization and data type, and reconfiguring the transmission block size using the second adjustment strategy; when it is determined that the timeout period is greater than the preset time, determining a data packet retransmission strategy based on the packet loss type, and reconfiguring the retransmission data packets of the adaptive data transmission channel according to the data packet retransmission strategy.

[0171] Furthermore, in an exemplary embodiment, the data synchronization device also includes a first switching module, which is used to: when the shared memory channel and the adaptive data transmission channel have been successfully created, at least obtain the cache hit rate and access delay of the shared memory channel, and obtain the bandwidth utilization of the adaptive data transmission channel; when the cache hit rate of the shared memory channel is lower than a first threshold for multiple consecutive times and the access delay is greater than the first threshold, switch from the shared memory channel to the adaptive data transmission channel; when the bandwidth utilization is lower than a second threshold and the remaining resources of the shared memory channel are greater than the second threshold, switch from the adaptive data transmission channel to the shared memory channel.

[0172] Furthermore, in an exemplary embodiment, the data synchronization device also includes a second switching module, which is used to: obtain the current status of the shared memory channel and the current status of the adaptive data transmission channel when both the shared memory channel and the adaptive data transmission channel have been successfully created; switch from the shared memory channel to the adaptive data transmission channel when the current status of the shared memory channel indicates that a memory leak event or an access conflict event has occurred in the shared memory channel; and switch from the adaptive data transmission channel to the shared memory channel when the current status of the adaptive data transmission channel indicates that a batch packet loss event or a link interruption event has occurred in the adaptive data transmission channel.

[0173] Furthermore, in an exemplary embodiment, the data synchronization device also includes a third synchronization module, which is used to: obtain the cache hit rate, delay compliance rate and memory usage rate monitored in real time; determine the first availability score of the shared memory channel based on a first set of weight coefficients and a weighted sum value between the cache hit rate, delay compliance rate and memory usage rate, wherein the first set of weight coefficients includes a first weight corresponding to the cache hit rate, a first weight corresponding to the delay compliance rate and a first weight corresponding to the memory usage rate; determine the adaptive data transmission efficiency score based on a second set of weight coefficients and a weighted sum value between the cache hit rate, delay compliance rate and memory usage rate. a second availability score for the data transmission channel, wherein the second set of weight coefficients includes a second weight corresponding to the cache hit rate, a second weight corresponding to the delay compliance rate, and a second weight corresponding to the memory usage rate; when it is determined that the difference between the first availability score and the second availability score is higher than a preset difference and lasts for at least two monitoring cycles, determining a target channel from the shared memory channel and the adaptive data transmission channel, wherein the target channel is a channel with a larger value between the first availability score and the second availability score; and synchronizing data between the client and the object storage device through the target channel.

[0174] Furthermore, in an exemplary embodiment, the data synchronization device also includes a fourth switching module, which is used to: after the shared memory channel and the adaptive data transmission channel are successfully created, for the case where the client and the object storage device are located in the same node, synchronize data between the client and the object storage device through the shared memory channel; when it is determined that the real-time availability score of the shared memory channel is lower than a preset score, switch from the shared memory channel to the adaptive data transmission channel; for the case where the client and the object storage device are located in different nodes, preferentially synchronize data between the client and the object storage device through the adaptive data transmission channel; when it is detected that the client and the object storage device are temporarily deployed on the same node, switch from the adaptive data transmission channel to the shared memory channel.

[0175] Furthermore, in an exemplary embodiment, the second synchronization module is further configured to determine a queue depth parameter of the adaptive data transmission channel according to the following formula:

[0176] ;

[0177] Wherein, Optimal_Queue_Depth is a queue depth parameter, the BDP is the bandwidth-delay product of the physical server, the Block_Size is the size of a data block to be sent by the object storage device, MAX_QUEUE_DEPTH is the maximum queue depth, α is an adjustment coefficient, and Workload_Factor is a workload factor;

[0178] The adaptive data transmission channel is configured according to the queue depth parameter.

[0179] Furthermore, in an exemplary embodiment, the data synchronization device also includes a calculation module, which is used to: before any object storage device in the physical server performs the erasure code calculation, detect whether the calculation result of the erasure code calculation is hit in the shared memory cache of the physical server; if the shared memory cache hits the calculation result, directly obtain the calculation result from the shared memory cache; if the shared memory cache does not hit the calculation result, instruct any object storage device to perform the erasure code calculation and store the obtained calculation result in the shared memory cache.

[0180] Furthermore, in an exemplary embodiment, the calculation module is further configured to determine a hit rate of the shared memory cache at a current moment according to the following formula.

[0181] ;

[0182] Among them, Cache_Hit_probability is the hit rate of the shared memory cache at the current moment, λ is the time decay coefficient, μ is the access frequency coefficient, △t is the time interval between the last access time of the shared memory cache and the current moment, and N is the number of accesses to the shared memory cache in the historical time period; wherein, when the hit rate is greater than the third threshold, it is determined that the calculation result is hit in the shared memory cache, and when the hit rate is less than the third threshold, it is determined that the calculation result is not hit in the shared memory cache.

[0183] Furthermore, in an exemplary embodiment, the data synchronization device also includes an elimination module, which is used to: obtain the time interval between the last time the shared memory cache was accessed and the current time, and obtain the number of times the shared memory cache was accessed within a historical time period; when the time interval is greater than a fourth threshold and the number of accesses is less than the fourth threshold, eliminate the shared memory cache and determine a new shared memory cache from the physical server based on an election algorithm.

[0184] Furthermore, in an exemplary embodiment, the data synchronization device also includes a sending module, which is used to: detect whether there is a difference in data versions between any two object storage devices in the physical server based on the vector clock; determine the difference value of the data version difference when there is a difference in data versions between any two object storage devices in the physical server; determine the incremental data between the data versions of the any two object storage devices when the difference value is less than a fifth threshold, wherein the incremental data is the difference data of the first object storage device relative to the second object storage device, and the any two object storage devices include: the first object storage device and the second object storage device; send the incremental data to the second object storage device to make the data versions of the first object storage device and the second object storage device consistent.

[0185] Furthermore, in an exemplary embodiment, the sending module is further configured to determine the incremental data using the following formula:

[0186] ;

[0187] Wherein, Sync_Data_Size represents the incremental data, Base_Size is the size of the base data block, β is the version difference coefficient, Version_Diff is the number of version differences, γ is the compression rate determined according to data characteristics, and Compression_Ratio is the compression ratio.

[0188] Furthermore, in an exemplary embodiment, the data synchronization device also includes a repair module for: determining the repair priority of the at least two object storage devices when it is detected that there are faults in at least two object storage devices in the physical server; and performing a repair task on the third object storage device with the highest repair priority.

[0189] Furthermore, in an exemplary embodiment, the repair module is further configured to determine the repair priorities of the at least two object storage devices using the following formula:

[0190] ;

[0191] Where repair_priority is the repair priority, importance is the data importance factor of the corresponding object storage device, failure_risk is the failure risk factor of the corresponding object storage device, age is the data inconsistency time of the corresponding object storage device, and decay_rate is the decay rate.

[0192] Furthermore, in an exemplary embodiment, the repair module is also used to: obtain the data blocks required for repairing the third object storage device from the replica pool or erasure code pool of the physical server; when the third object storage device receives the required data blocks, verify the required data blocks, and confirm that the repair task of the third object storage device is completed after the verification passes.

[0193] For the description of the features in the embodiment corresponding to the data synchronization device, please refer to the relevant description of the embodiment corresponding to the data synchronization method, and no further details will be given here.

[0194] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data synchronization method embodiments.

[0195] Optionally, Figure 4 : is a block diagram of the computer system structure of the electronic device according to the embodiment of the present application. Figure 4As shown, computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage 408 into random access memory (RAM) 803. RAM 403 also stores various programs and data required for system operation. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output interface 405 (I / O interface) is also connected to bus 404.

[0196] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 408 including a hard disk; and a communication section 409 including a network interface card such as a local area network card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read from the removable media can be installed in the storage section 408 as needed.

[0197] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0198] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data synchronization method embodiments when running.

[0199] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0200] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data synchronization method embodiments are implemented.

[0201] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data synchronization method embodiments are implemented.

[0202] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] The above is a detailed introduction to a data synchronization method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data synchronization method, characterized in that: include: Detect the location relationship between the client and the object storage device; When the location relationship indicates that the client and the object storage device are located on the same node, creating a shared memory channel, and synchronizing data between the client and the object storage device through the shared memory channel; When the location relationship indicates that the client and the object storage device are located at different nodes, configuring an adaptive data transmission channel according to a network performance parameter of the physical server, and synchronizing data between the client and the object storage device through the adaptive data transmission channel, wherein both the client and the object storage device are located at the physical server; The method further comprises: When both the shared memory channel and the adaptive data transmission channel have been successfully created, obtaining a current state of the shared memory channel and a current state of the adaptive data transmission channel; When the current state of the shared memory channel indicates that a memory leak event or an access conflict event occurs in the shared memory channel, switching from the shared memory channel to the adaptive data transmission channel; When the current state of the adaptive data transmission channel indicates that a batch packet loss event or a link interruption event occurs on the adaptive data transmission channel, the adaptive data transmission channel is switched to the shared memory channel.

2. The data synchronization method according to claim 1, characterized in that: Create a shared memory channel, including: Determining whether the memory capacity of the physical server is greater than a preset capacity, wherein the preset capacity represents a capacity corresponding to a minimum memory requirement for creating the shared memory channel; When the memory capacity is greater than the preset capacity, obtaining a shared memory identifier of the shared memory channel, wherein the shared memory identifier is generated based on a node identifier of the same node and a storage pool identifier corresponding to the same node; The shared memory channel is determined according to the channel indicated by the shared memory identifier.

3. The method according to claim 1, characterized in that The network performance parameters include at least one of the following: queue depth, transmission block size, timeout period, and configuring an adaptive data transmission channel based on the network performance parameters of the physical server includes at least one of the following: determining a transmission delay and a queue filling rate of the physical server, determining a first adjustment strategy for the queue depth according to the transmission delay and the queue filling rate, and reconfiguring the queue depth using the first adjustment strategy; Determining a bandwidth utilization and a data type of the physical server, determining a second adjustment strategy for the transmission block size according to the bandwidth utilization and the data type, and reconfiguring the transmission block size using the second adjustment strategy; When it is determined that the timeout period is greater than the preset time, a data packet retransmission strategy is determined according to the packet loss type, and the retransmission data packet of the adaptive data transmission channel is reconfigured according to the data packet retransmission strategy.

4. The method according to claim 1, wherein The method further comprises: When both the shared memory channel and the adaptive data transmission channel have been successfully created, obtaining at least a cache hit rate and an access delay of the shared memory channel, and obtaining a bandwidth utilization rate of the adaptive data transmission channel; When the cache hit rate of the shared memory channel is lower than the first threshold for multiple consecutive times and the access delay is greater than the first threshold, switching from the shared memory channel to the adaptive data transmission channel; When the bandwidth utilization is lower than a second threshold and the remaining resources of the shared memory channel are greater than the second threshold, switching is performed from the adaptive data transmission channel to the shared memory channel.

5. The method according to claim 1, wherein The method further comprises: Obtain real-time monitoring of cache hit rate, latency compliance rate, and memory usage; Determining a first availability score of the shared memory channel based on a first set of weight coefficients and a weighted sum of the cache hit rate, the latency compliance rate, and the memory usage rate, wherein the first set of weight coefficients includes a first weight corresponding to the cache hit rate, a first weight corresponding to the latency compliance rate, and a first weight corresponding to the memory usage rate; determining a second availability score of the adaptive data transmission channel based on a second set of weight coefficients and a weighted sum of the cache hit rate, the delay compliance rate, and the memory usage rate, wherein the second set of weight coefficients includes a second weight corresponding to the cache hit rate, a second weight corresponding to the delay compliance rate, and a second weight corresponding to the memory usage rate; When it is determined that the difference between the first availability score and the second availability score is greater than a preset difference and lasts for at least two monitoring cycles, determining a target channel from the shared memory channel and the adaptive data transmission channel, wherein the target channel is a channel having a larger value between the first availability score and the second availability score; Data synchronization is performed between the client and the object storage device through the target channel.

6. The method according to claim 1, characterized in that The method further comprises: After both the shared memory channel and the adaptive data transmission channel are successfully created, in the case where the client and the object storage device are located on the same node, data synchronization between the client and the object storage device is performed through the shared memory channel; if it is determined that the real-time availability score of the shared memory channel is lower than a preset score, switching from the shared memory channel to the adaptive data transmission channel; In the case where the client and the object storage device are located in different nodes, data synchronization between the client and the object storage device is preferentially performed through the adaptive data transmission channel; when it is detected that the client and the object storage device are temporarily deployed on the same node, the adaptive data transmission channel is switched to the shared memory channel.

7. The method according to claim 1, characterized in that Configure adaptive data transmission channels based on the network performance parameters of the physical server, including: The queue depth parameter of the adaptive data transmission channel is determined according to the following formula: ； Wherein, Optimal_Queue_Depth is a queue depth parameter, the BDP is the bandwidth-delay product of the physical server, the Block_Size is the size of the data block to be sent by the object storage device, MAX_QUEUE_DEPTH is the maximum queue depth, α is an adjustment coefficient, and Workload_Factor is a workload factor; The adaptive data transmission channel is configured according to the queue depth parameter.

8. The method according to claim 1, characterized in that The method further comprises: Before any object storage device in the physical server performs erasure code calculation, detecting whether a calculation result of the erasure code calculation is hit in the shared memory cache of the physical server; When the shared memory cache hits the calculation result, directly obtaining the calculation result from the shared memory cache; In the case that the shared memory cache does not hit the calculation result, instruct any one of the object storage devices to perform erasure code calculation and store the obtained calculation result in the shared memory cache.

9. The method according to claim 8, characterized in that Detecting whether a calculation result of the erasure code calculation is hit in the shared memory cache of the physical server includes: The hit rate of the shared memory cache at the current moment is determined according to the following formula: ； Wherein, Cache_Hit_probability is the hit rate of the shared memory cache at the current moment, λ is the time decay coefficient, μ is the access frequency coefficient, Δt is the time interval between the last access time of the shared memory cache and the current moment, and N is the number of accesses to the shared memory cache in the historical time period; Wherein, when the hit rate is greater than a third threshold, it is determined that the calculation result is hit in the shared memory cache; when the hit rate is less than the third threshold, it is determined that the calculation result is not hit in the shared memory cache.

10. The method according to claim 9, characterized in that The method further comprises: Obtaining the time interval between the last time the shared memory cache was accessed and the current time, and obtaining the number of times the shared memory cache was accessed within a historical time period; When the time interval is greater than a fourth threshold and the number of accesses is less than the fourth threshold, the shared memory cache is eliminated, and a new shared memory cache is determined from the physical server based on an election algorithm.

11. The method according to claim 1, wherein The method further comprises: Detecting, based on a vector clock, whether there are differences in data versions between any two object storage devices in the physical server; In the case that there is a difference in data versions between any two object storage devices in the physical server, determining a difference value of the difference in data versions; When the difference value is less than a fifth threshold, determining incremental data between the data versions of the arbitrary two object storage devices, wherein the incremental data is difference data of a first object storage device relative to a second object storage device, and the arbitrary two object storage devices include: the first object storage device and the second object storage device; The incremental data is sent to the second object storage device to make the data versions of the first object storage device and the second object storage device consistent.

12. The method according to claim 11, characterized in that Determining incremental data between data versions of any two object storage devices includes: The incremental data is determined by the following formula: ； Wherein, Sync_Data_Size represents the incremental data, Base_Size is the size of the base data block, β is the version difference coefficient, Version_Diff is the number of version differences, γ is the compression rate determined according to data characteristics, and Compression_Ratio is the compression ratio.

13. The method according to claim 1, wherein The method further comprises: In a case where it is detected that at least two object storage devices in the physical server are faulty, determining repair priorities of the at least two object storage devices; The repair task is performed on the third object storage device with the highest repair priority.

14. The method according to claim 13, characterized in that Determining the repair priorities of the at least two object storage devices includes: The repair priorities of the at least two object storage devices are determined by the following formula: ； Where repair_priority is the repair priority, importance is the data importance factor of the corresponding object storage device, failure_risk is the failure risk factor of the corresponding object storage device, age is the data inconsistency time of the corresponding object storage device, and decay_rate is the decay rate.

15. The method according to claim 13, characterized in that Perform a repair task on the third object storage device with the highest repair priority, including: Obtaining data blocks required for repairing the third object storage device from the replica pool or erasure code pool of the physical server; When the third object storage device receives the required data block, the required data block is verified, and after the verification passes, it is confirmed that the repair task of the third object storage device is completed.

16. A data synchronization device, characterized in that: include: A detection module, used to detect the position relationship between the client and the object storage device; a first synchronization module, configured to, when the position relationship indicates that the client and the object storage device are located on the same node, create a shared memory channel and synchronize data between the client and the object storage device through the shared memory channel; a second synchronization module, configured to, when the location relationship indicates that the client and the object storage device are located at different nodes, configure an adaptive data transmission channel according to a network performance parameter of the physical server, and synchronize data between the client and the object storage device through the adaptive data transmission channel, wherein both the client and the object storage device are located at the physical server; The first switching module is used to obtain the current status of the shared memory channel and the current status of the adaptive data transmission channel when both the shared memory channel and the adaptive data transmission channel have been successfully created; switch from the shared memory channel to the adaptive data transmission channel when the current status of the shared memory channel indicates that a memory leak event or an access conflict event has occurred in the shared memory channel; and switch from the adaptive data transmission channel to the shared memory channel when the current status of the adaptive data transmission channel indicates that a batch packet loss event or a link interruption event has occurred in the adaptive data transmission channel.

17. An electronic device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the data synchronization method according to any one of claims 1 to 15 when executing the computer program.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 15 are implemented.

Images