Data synchronization method, storage system, device, and storage medium

Abstract

The application provides a data synchronization method, a storage system, equipment and a storage medium, and relates to the field of storage. The storage system comprises a first computing node, a second computing node and a shared storage device. When the first computing node flushes the obtained primary copy into a data segment in the memory, the first computing node copies a first part of data of the data segment to the second computing node through network transmission, and stores the data segment into the shared storage device when preset storage conditions are met. The second computing node processes and stores the received first part of data, and reads and loads a second part of data of the data segment from the shared storage device when it is determined that the second part of data is not stored. The first part of data and the second part of data are secondary copies of the primary copy. In the technical scheme, in the storage system architecture of separation of computing and storage, the primary and secondary copy synchronization is realized by using two synchronization links of the shared storage device and the network transmission, and the data synchronization efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of storage, in particular to a data synchronization method, a storage system, a device and a storage medium. BACKGROUND

[0002] In the storage process of data, in order to ensure data security, a distributed storage system generally uses multiple copies to store data. One copy of the multiple copies is used as a master copy, and the other copies are used as slave copies. Since the multiple copies are located at different positions, even if a fault occurs in a copy, data will not be lost. Therefore, how to realize data synchronization of multiple copies is a problem to be solved.

[0003] At present, data synchronization of master and slave copies can be realized based on network transmission. That is, after a certain computing node receives data written by a user and flushes the data into a master copy, on one hand, the master copy is written to a local disk, and on the other hand, the master copy is synchronized to other computing nodes through network transmission to obtain a slave copy, and the other computing nodes store the slave copy to the local disk, thereby realizing data synchronization of master and slave copies. However, in this method, data calculation and storage are integrated, and there are problems of low data synchronization efficiency, difficulty in expanding and shrinking the capacity of the copy, and poor flexibility. SUMMARY

[0004] The present application provides a data synchronization method, a storage system, a device and a storage medium to realize the separation of data calculation and storage, and solve the problems of difficulty in expanding and shrinking the capacity of the copy, low data synchronization efficiency and poor flexibility.

[0005] In a first aspect, the present application provides a data synchronization method applied to a storage system, wherein the storage system comprises a first computing node, a second computing node and a shared storage device, and the method comprises:

[0006] When the first computing node flushes the obtained master copy into a data segment in memory, the first computing node copies a first part of data in the data segment to the second computing node through network transmission;

[0007] The second computing node processes and stores the received first part of data;

[0008] When a preset storage condition is met, the first computing node stores the data segment in the memory to the shared storage device, and the storage condition comprises that a total size of the data segment in the memory is greater than a capacity threshold, and / or a time interval from the previous storage is greater than an interval threshold;

[0009] When the second computing node determines that the second part of the data segment is not stored, it reads and loads the second part of the data from the shared storage device, wherein the first part of the data and the second part of the data are slave copies of the primary copy.

[0010] Secondly, this application provides a data synchronization method applied to a first computing node, the method comprising:

[0011] When the acquired primary copy is refreshed into a data segment in memory, the first part of the data segment is copied to the second computing node via network transmission.

[0012] When the preset storage conditions are met, the data segments in the memory are persistently stored in the shared storage device. The shared storage device is used for the second computing node to read and load the second part of the data in the unstored data segments. The storage conditions include the total size of the data segments in the memory being greater than the capacity threshold, and / or the time interval since the last storage being greater than the interval threshold.

[0013] Thirdly, this application provides a data synchronization method applied to a second computing node, the method comprising:

[0014] Receive the first portion of synchronized data from the first computing node;

[0015] The first part of the data is verified and merged to obtain the target data segment;

[0016] If it is determined that the second part of the data is not stored, the second part of the data is read from and loaded from the shared storage device, which is used to store the primary copy obtained by the first computing node, and the first part of the data and the second part of the data are secondary copies of the primary copy.

[0017] Fourthly, this application provides a data synchronization method applied to a shared storage device, the method comprising:

[0018] The first computing node receives a data segment to be stored. This data segment is sent by the first computing node when preset storage conditions are met. The storage conditions include that the total size of the data segments in memory is greater than a capacity threshold, and / or that the time interval since the last storage is greater than an interval threshold.

[0019] Based on the identifier of the first computing node, the data segment is stored in the target location;

[0020] Receive a data read request from the second computing node, the data read request being issued by the second computing node when it determines that the second part of the data is not stored;

[0021] The second part of the data is determined based on the data read request and transmitted to the second computing node.

[0022] Fifthly, this application provides a storage system, which includes a first computing node, a second computing node, and a shared storage device, wherein:

[0023] When the first computing node refreshes the acquired primary copy into a data segment in memory, it copies the first part of the data segment to the second computing node via network transmission.

[0024] The second computing node processes and stores the received first portion of data;

[0025] When the preset storage conditions are met, the first computing node stores the data segments in the memory to the shared storage device. The storage conditions include that the total size of the data segments in the memory is greater than the capacity threshold, and / or that the time interval since the last storage is greater than the interval threshold.

[0026] When the second computing node determines that the second part of the data segment is not stored, it reads and loads the second part of the data from the shared storage device, wherein the first part of the data and the second part of the data are slave copies of the primary copy.

[0027] In a sixth aspect, this application provides an electronic device including a processor for invoking a computer program (also referred to as code or instructions) to enable the electronic device to implement the technical solutions of the first computing node or the second computing node as described in the first to fourth aspects, or to implement the technical solutions of the shared storage device.

[0028] In a seventh aspect, this application provides a computer-readable storage medium including a computer program that, when run on a computer, causes the computer to execute the technical solutions of the first computing node or the second computing node or to implement the technical solutions of the shared storage device in the first to fourth aspects.

[0029] Eighthly, this application provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when executed, causes a computer to execute the technical solution of the first computing node or the technical solution of the second computing node or to implement the technical solution of the shared storage device in the first to fourth aspects.

[0030] In this embodiment, the storage system includes a first computing node, a second computing node, and a shared storage device. When the first computing node refreshes the acquired primary copy into data segments in memory, it copies a first portion of the data from the data segments to the second computing node via network transmission. Furthermore, if the total size of the data segments in memory exceeds a capacity threshold and / or the time interval since the last storage exceeds an interval threshold, the first computing node can persistently store the data segments in memory to the shared storage device. Correspondingly, the second computing node can process and store the received first portion of data, and upon determining that a second portion of data has not been stored, it reads and loads the second portion of data from the shared storage device. The first and second portions of data are slave copies of the primary copy. In this technical solution, under a storage system architecture that separates computing and storage, data synchronization between primary and slave copies is achieved using two synchronization links: a shared storage device and / or network transmission, improving the data synchronization efficiency between primary and slave copies. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a data synchronization architecture based on shared storage devices;

[0032] Figure 2 This is a schematic diagram of a data synchronization architecture based on network transmission;

[0033] Figure 3 This is a schematic block diagram of the storage system provided in the embodiments of this application;

[0034] Figure 4 This is an interactive schematic diagram of the data synchronization method provided in the first embodiment of this application;

[0035] Figure 5 This is an interactive schematic diagram of the data synchronization method provided in the second embodiment of this application;

[0036] Figure 6 A schematic block diagram of a computing node provided in an embodiment of this application;

[0037] Figure 7 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0039] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0040] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion.

[0041] The following is a brief explanation of the relevant terms used in the embodiments of this application.

[0042] 1. Primary Replica (Read-Write Replica): In a distributed storage system, data is redundantly stored in multiple replicas for backup. One of these replicas serves as the primary replica, or read-write replica, providing write and read capabilities. The compute node hosting the primary replica can synchronize it to other compute nodes to generate secondary replicas through physical or logical replication.

[0043] 2. From replica (read-only replica): In a distributed storage system, data is redundantly stored in multiple replicas for backup. The replica that only provides read capability is called a slave replica, or read-only replica.

[0044] 3. Shared storage devices: Unlike local disk storage, shared storage devices are distributed file systems that allow multiple machines on a network to read and write the same data. Furthermore, shared storage devices can be understood as storage devices that can permanently store data, such as hard disks.

[0045] 4. Physical Replication: In the master-slave replication model, after the master replica is written to the first compute node, an index is built on the first compute node, and then the entire built index file is synchronized to the second compute node to generate a slave replica, avoiding duplicate building on the second compute node.

[0046] Optionally, in the embodiments of this application, building an index on the first computing node can be understood as refreshing the main copy newly written by the user into a data segment.

[0047] 5. Logical Replication: In the master-slave replication model, data is written to the first compute node and simultaneously to the second compute node. The first and second compute nodes independently execute their respective index building processes, thereby achieving data synchronization between the master and slave replicas.

[0048] Figure 1 This is a schematic diagram of a data synchronization architecture based on shared storage devices. For example... Figure 1As shown in the diagram, the architecture may include at least two compute nodes, at least two local disks, and a shared storage device 10. The compute nodes and local disks correspond to each other.

[0049] Optional, refer to Figure 1 As shown in the diagram, this architecture diagram is illustrated with n compute nodes (compute node 1 to compute node n) and n local disks (local disk 1 to local disk n). It is understood that in practical applications, the specific number of compute nodes and local disks shown in the architecture diagram is not limited; they can be determined according to actual needs, and will not be elaborated upon here.

[0050] Understandable, Figure 1 In the architecture diagram shown, compute nodes and local disks with corresponding relationships reside on the same device. For example, compute node 1 and local disk 1 are on the same device, compute node 2 and local disk 2 are on the same device, and compute node n and local disk n are on the same device.

[0051] In practical applications, users can write data to any one of the at least two computing nodes mentioned above. The computing node to which the data is written becomes the computing node that carries the primary replica, i.e., the read-write node, thus providing write and read capabilities. Correspondingly, one or more of the other computing nodes become the computing nodes that carry the secondary replica.

[0052] During the data synchronization process between master and slave replicas, after compute node 1 obtains the data written by the user, it first generates an index file for the data, and then writes the index file to local disk 1. Subsequently, it periodically writes the data in local disk 1 to shared storage device 10. Correspondingly, compute nodes 2 to n can read and load the data stored in shared storage device 1 by compute node 1 from shared storage device 10, thereby achieving data synchronization.

[0053] Understandable Figure 1 The illustrated architecture diagram is based on a data synchronization method using shared storage devices. In practical applications, to accelerate the data write rate and buffer the pressure of data writes on the shared storage device, after compute node 1 obtains the data written by the user, it first writes the data to its local disk, and then periodically writes it to the shared storage device. This process, where other compute nodes can only share the data in compute node 1 after it has been stored in the shared storage device, results in a long data synchronization link between compute nodes, leading to long synchronization latency between master and slave replicas, low data synchronization efficiency, and poor data write performance. Furthermore, since both master and slave replicas need to pass through the shared storage device for reading and writing, read / write amplification occurs.

[0054] Optional, Figure 2This is a schematic diagram of a data synchronization architecture based on network transmission. For example... Figure 2 As shown in the diagram, this architecture may include at least two compute nodes and at least two local disks. The compute nodes and local disks correspond to each other.

[0055] Similarly, in Figure 2 The architecture diagram shown is illustrated with n compute nodes (compute node 1 to compute node n) and n local disks (local disk 1 to local disk n). It should be understood that in practical applications, the specific number of compute nodes and local disks shown in the architecture diagram is not limited; they can be determined based on actual needs, and will not be elaborated upon here.

[0056] In practical applications, users can write data to any one of the at least two computing nodes mentioned above, becoming a computing node that carries the primary replica, i.e., a read-write node, thereby providing write and read capabilities. Correspondingly, one or more of the other computing nodes can become computing nodes that carry the secondary replica.

[0057] Understandable, Figure 2 In the architecture diagram shown, compute nodes and local disks with corresponding relationships reside on the same device. For example, compute node 1 and local disk 1 are on the same device, compute node 2 and local disk 2 are on the same device, and compute node n and local disk n are on the same device.

[0058] During the data synchronization process between master and slave replicas, after compute node 1 receives the data written by the user, it first generates an index file for the data, then writes the index file to local disk 1. Simultaneously, it synchronizes the index file to other compute nodes, which then store the index file to their respective local disks, thus achieving data synchronization between master and slave replicas. In this way, each compute node can read data from its corresponding local disk to provide query services.

[0059] Depend on Figure 2 As shown in the architecture diagram, in the network-based data synchronization scheme, all index files of compute node 1 are sent to other compute nodes via the network. This results in a large amount of network data copying during failover and scaling, leading to long recovery times. In other words, network-based data synchronization integrates storage and compute, failing to take advantage of storage-compute separation and exhibiting difficulties in scaling compute nodes and poor flexibility.

[0060] In view of this, this application provides a data synchronization method applied to a storage system, which includes a first computing node, a second computing node, and a shared storage device. When the first computing node refreshes the acquired primary copy into a data segment in memory, it copies a first portion of the data segment to the second computing node via network transmission. When preset storage conditions are met, the data segment is stored in the shared storage device. The second computing node processes and stores the received first portion of the data, and when it determines that a second portion of the data segment has not been stored, it reads and loads the second portion of the data from the shared storage device. The first and second portions of the data are slave copies of the primary copy. The process of the first computing node refreshing the acquired primary copy into a data segment in memory can be interpreted as the first computing node caching the primary copy in a buffer and periodically refreshing it to the NRT cache. This technical solution achieves primary-slave copy data synchronization through a hybrid synchronization method of shared storage device and network transmission, improving data synchronization efficiency and flexibility, and fully utilizing bandwidth resources.

[0061] Furthermore, in the technical solution provided in this application, after the primary copy is refreshed into data segments, on the one hand, the first computing node and the second computing node can synchronize data through physical replication; on the other hand, for the data segments that are not synchronized to the second computing node, the second computing node can directly read and load them from the shared storage device, thus realizing the separation of computing and storage, and having the characteristics of elastic scaling (storage resources and computing resources can be expanded and contracted independently) and good flexibility.

[0062] In the compute-storage separation architecture, master-slave data synchronization is achieved using two synchronous links: shared storage devices and network transmission. This improves data write performance and reduces the port pressure on shared storage devices. Compared to master-slave synchronization that relies solely on shared storage devices, point-to-point network transmission reduces network hops and lowers master-slave data latency.

[0063] It should be noted that the data processing method provided in this application embodiment can be applied to a storage system. Before introducing the data synchronization method provided in this application embodiment, the storage system provided in this application embodiment will first be explained.

[0064] Figure 3 This is a schematic block diagram of a storage system provided in an embodiment of this application. The storage system can at least be used to provide data storage services and query services, such as... Figure 3 As shown, the storage system 300 includes a first computing node 31, a second computing node 32, and a shared storage device 33.

[0065] For example, Figure 3The storage system is explained with one first computing node and one second computing node. In practical applications, there may be multiple second computing nodes, and this embodiment does not limit the number of them.

[0066] exist Figure 3 In the storage system shown, the first computing node 31 may include a read / write engine 311 and a near real-time (NRT) cache 312.

[0067] The read / write engine 311 can include a buffer 3110. After obtaining data written by the user (i.e., the primary replica), the read / write engine 311 first writes the primary replica to buffer 3110, and then periodically refreshes the primary replica in buffer 3110 into data segments in the NRT cache 312. Only then can the primary replica written by the user become searchable. In other words, there is a small delay between the primary replica being written and becoming searchable (approximately 1 second; for Elasticsearch-based search and analysis, this can be reduced to the second level). For example, refer to... Figure 3 As shown, when the primary copy in buffer 3110 is flushed to a data segment in NRT cache 312, such as data segment A, the read / write engine 311 can then read from NRT cache 312.

[0068] It is understood that, in the embodiments of this application, the primary replica written to the first computing node 31 is directly written to buffer 3110, and then, after the index building process, forms a file that can be queried in NRT cache 312.

[0069] Optionally, in one possible implementation of this application, when the first computing node 31 finishes refreshing the received data into a data segment, data synchronization between the first computing node 31 and the second computing node 32 can be triggered. That is, the first computing node 31 can synchronize the data segment A stored in the NRT cache 312 to the NRT cache 322 in the second computing node 32 through physical replication. In this way, after the second computing node 32 verifies and loads the received data segment A, it can provide query services. For example, see [link to relevant documentation]. Figure 3 As shown, the read-only engine 321 of the second compute node 32 can read data segment A from the NRT cache 322.

[0070] Optionally, in embodiments of this application, the first computing node 31 may also flush and persist the data segment in the NRT cache 312 to the shared storage device 33 when the preset storage conditions are met. For example, the data segment in the NRT cache 312 may be persisted as a disk segment, such as data segment B in the shared storage device 33.

[0071] As an example, in order to control the capacity of the NRT cache 312 in the first compute node 31, the first compute node 31 can flush and persist the data segments in the NRT cache 312 to the shared storage device 33 when the total size of the data segments in the NRT cache 312 exceeds the capacity threshold.

[0072] As another example, in order to store the primary copy of the first compute node 31 to the shared storage device 33 in a timely manner, the first compute node 31 can periodically flush and persist the data segments in the NRT cache 312 to the shared storage device 33. That is, when the time interval since the last storage is greater than the interval threshold, the first compute node 31 can flush and persist the data segments in the NRT cache 312 to the shared storage device 33.

[0073] In the embodiments of this application, persistent storage process can be understood as the process of storing data to a storage device that can permanently store the data.

[0074] Understandably, once the data segment is persisted to the shared storage device 33, it can be read and loaded by the first compute node 31 and the second compute node 32. For example, see... Figure 3 As shown, the read / write engine 311 of the first computing node 31 and the read-only engine 321 of the second computing node 32 can both read data segment B from the shared storage device 33.

[0075] In practical applications, several data segments can form a shard. Correspondingly, the data segment corresponding to the primary replica can also be called the primary shard. The slave replica synchronized to the second computing node 32 can be called the replica shard. The replica shard is a replica of the primary shard and is responsible for fault tolerance and bearing the load of read requests.

[0076] In the embodiments of this application, the shared storage device 33 can be read and loaded by different computing nodes, thereby realizing the separation of computing and reading / writing. While ensuring data security, it can also meet the purpose of independent expansion and contraction, thus improving flexibility.

[0077] Optional, refer to Figure 3As shown, after the first compute node 31 obtains the data written by the user, it updates the buffer 3110 with the newly written data and writes a log file (translog) to the shared storage device 33. This is so that when the first compute node 31 is lost and fails, the second compute node 32 can cooperate with the translog to be promoted to the new first compute node.

[0078] Optionally, after the log file is written, the first compute node 31 can return a write success response to the user. However, the newly written data is not yet visible at this time and needs to be refreshed periodically to flush the primary replica in buffer 3110 to the segments in NRT cache 312 before it can be queried.

[0079] Optionally, the read / write engine 311 can also periodically perform a merge process to combine smaller segments into larger segments in order to control the total number of segments in the NRT cache and avoid query performance loss.

[0080] Optionally, the read / write engine 311 can also periodically perform a persistence process (flush) to persist the segment in memory, i.e., write it to the shared storage device 33, and reclaim the translog written before the segment is persisted, so as to reduce unnecessary memory usage.

[0081] Understandable Figure 3 The storage system shown is merely an example. In practical applications, the storage system may include other components, which will not be elaborated here. Furthermore, in real-world scenarios, the first computing node and the second computing node are distinguished based on whether they have the capability to write data. Typically, a computing node is considered a first computing node for a given set of data, but may be considered a second computing node for a set of data different from the first data. This embodiment of the application does not limit this distinction.

[0082] Optionally, after understanding the composition and function of the storage system, the following combination Figure 3 The storage system shown illustrates the data synchronization method provided in the embodiments of this application. The following embodiments are explained using the information interaction between the first computing node, the second computing node, and the shared storage device in the storage system.

[0083] For example, Figure 4 This is an interactive schematic diagram of the data synchronization method provided in the first embodiment of this application. This data synchronization method is applied to... Figure 3 The storage system shown, such as Figure 4As shown, the data synchronization method may include the following steps:

[0084] S401. The first compute node writes the received primary replica into memory and periodically refreshes the primary replica in memory into data segments.

[0085] In practical applications, each computing node in a storage system can serve as a read / write node for data. It can be understood that when a computing node acts as a read / write node for a particular piece of data, it becomes the primary computing node for that data, and the other computing nodes become secondary computing nodes carrying that data.

[0086] For example, when the first compute node obtains the primary copy written by the user, the read / write engine of the first compute node writes the primary copy into memory and periodically refreshes the primary copy in memory based on a preset period to generate data segments.

[0087] Optionally, in embodiments of this application, memory may include a buffer and an NRT cache. The buffer is used for buffered data writing, and the NRT cache is used to hold queried files and subsequent physical copies. Accordingly, in this embodiment, the first computing node writes the received primary copy into memory and periodically refreshes the primary copy in memory into data segments, including: the first computing node writes the received primary copy into the buffer, and periodically refreshes the primary copy in the buffer into data segments in the NRT cache.

[0088] In one possible implementation of this application, taking the Lucene index as an example, the Lucene index consists of several segments, and each segment consists of several files. In this embodiment, the several files can be referred to as several data entries, that is, the Lucene index is written to the first computing node in the form of data.

[0089] It is understood that in the embodiments of this application, the data written by the user may also be called an index or a master copy, and the data segment may also be called a segment file; no limitation is made here.

[0090] For example, when a user writes a Lucene index to the first compute node, data is sequentially written to the read / write engine of that first compute node. The read / write engine then writes the newly acquired primary replica to the buffer, updating the buffer with the newly written data. Since the primary replica in the buffer is not yet visible, it needs to be refreshed into a segment in the NRT cache so that it can be queried promptly. However, to improve write performance, the read / write engine typically performs a data refresh process periodically. It is understood that the duration of the data refresh cycle can be set based on actual needs and is not limited here.

[0091] Optionally, in one possible implementation of this application embodiment, after the first computing node periodically refreshes the primary copy in memory into data segments, it can also periodically merge the data segments in memory. For example, the read / write engine of the first computing node can periodically execute a data segment merging process to merge smaller segments into larger segments, thereby controlling the total number of segments and avoiding query performance loss. For example, the first computing node periodically refreshes the data in memory to generate memory segments, and the background merge periodically merges smaller memory segments into larger memory segments.

[0092] Understandably, these segments are all cached in Figure 3 In the NRT cache of the illustrated embodiment.

[0093] S402. When the memory refresh process of the first computing node ends, the first part of the data in the data segment is copied to the second computing node through network transmission.

[0094] In the embodiments of this application, the first computing node can periodically perform a refresh process on the primary replica in the buffer, and after the refresh process ends and a data segment is generated, the data synchronization process of the primary second computing node can be triggered.

[0095] For example, after the refresh process is completed, the first compute node triggers an interaction process between the first and second compute nodes to synchronize the segment in the NRT cache of the first compute node to the NRT cache of the second compute node via physical replication. That is, the first part of the data segment in the NRT cache of the first compute node can be copied or replicated to the second compute node via network transmission.

[0096] It is understood that the network transmission in this embodiment is not limited to any particular form; it can be wired network transmission, wireless network transmission, or other forms, which will not be elaborated here.

[0097] S403, The second computing node processes and stores the received first part of the data.

[0098] Optionally, after receiving the first part of the data segment, the second computing node can verify, load, and link the received first part of the data, and provide query services to the outside world after obtaining the complete data.

[0099] S404. When the preset storage conditions are met, the first computing node stores the data segment in memory to the shared storage device.

[0100] The storage conditions include the total size of data segments in memory being greater than the capacity threshold, and / or the time interval since the last storage being greater than the interval threshold.

[0101] S405. When the second computing node determines the second part of the data in the unstored data segment, it reads and loads the second part of the data from the shared storage device.

[0102] The first and second parts of the data are slave copies of the primary copy.

[0103] In the embodiments of this application, the first computing node can also perform a data persistence process (flush process), for example, periodically or when the total size of the data segment is greater than the capacity threshold, write the data segment in memory to the shared storage device for persistence. That is, when the data segment is persisted to the shared storage device, the second computing node can directly read the shared storage device and load the required second part of the data when it determines that the second part of the data segment has not been stored.

[0104] Optionally, the second part of the data segment not stored by the second computing node may be data that has not been synchronized through the network in a timely manner, or data whose size is greater than a preset threshold; there is no limitation here.

[0105] For example, for a data segment buffered in the NRT cache, when it is determined that the total size of the data segments in memory is greater than the capacity threshold, persisting the data segments in memory to the shared storage device may include: when the first computing node determines that the total size of the data segments in the NRT cache is greater than the capacity threshold of the NRT cache, persisting the data segments in the NRT cache to the shared storage device.

[0106] Optionally, since the NRT cache in the replica has limited capacity, in practical applications, it is necessary to control the NRT cache capacity. For example, control the total size of data segments in the NRT cache. When the total size of data segments exceeds the preset capacity threshold, the first compute node can flush and persist the data segments in the NRT cache to the shared storage device and persist the segments in memory to disk segments.

[0107] Optionally, to avoid system failures, the first compute node can also periodically perform the process of persisting data segments to the shared storage device. For example, when the time interval since the last storage exceeds an interval threshold, the first compute node can persist data segments in the NRT cache to the shared storage device.

[0108] It is understood that the embodiments of this application are applied to scenarios where the storage layer of a Table Store multi-dimensional index is transformed from a local disk to a shared storage device. In this embodiment, data synchronization between the first computing node and the second computing node can be achieved through physical replication, that is, by constructing an index file (i.e., a data segment) on the first computing node, and then synchronizing the entire index file to the second computing node. In other words, the file in the memory of the first computing node can be directly transmitted to the second computing node via the network, saving the computing engine resources consumed by the second computing node from repeatedly constructing the index file.

[0109] Furthermore, in this embodiment, the index file on the first computing node can also be persisted to the shared storage device. This way, for files persisted to the shared storage device or large index files, the second computing node does not need to copy them and can directly read and load them from the shared storage device, thereby improving data synchronization efficiency while ensuring data security.

[0110] In one possible implementation of this application, the second computing node can receive synchronized first part data from the first computing node, verify and merge the first part data to obtain and store the target data segment. In addition, when it is determined that the second part data has not been stored, the second part data can also be read and loaded from the shared storage device. The shared storage device is used to store the master copy obtained by the first computing node, and the first part data and the second part data are slave copies of the master copy.

[0111] Accordingly, in the embodiments of this application, the shared storage device can receive a data segment to be stored from a first computing node, wherein the data segment is sent by the first computing node when a preset storage condition is met, the storage condition including the total size of the data segments in the memory being greater than a capacity threshold, and / or the time interval since the last storage being greater than an interval threshold; based on the identifier of the first computing node, the data segment is stored in a target location. In addition, the shared storage device can also receive a data read request from a second computing node, which is issued by the second computing node when it determines that the second part of the data has not been stored, the second part of the data is determined based on the data read request, and transmitted to the second computing node.

[0112] In the embodiments of this application, after the primary replica is refreshed into data segments, on the one hand, the first computing node and the second computing node can synchronize the first part of the data in the data segment through physical replication; on the other hand, for the second part of the data in the data segment that has not been synchronized to the second computing node, the second computing node can directly read and load it from the shared storage device. This technical solution achieves data synchronization between primary and secondary replicas through a hybrid synchronization method of shared storage devices and network transmission, improving data synchronization efficiency and fully utilizing bandwidth resources.

[0113] The data synchronization method provided in this application improves data writing performance and alleviates the input / output pressure on the shared storage device by transferring a portion of the data written to the shared storage device during the master-slave synchronization process to network transmission. This avoids the problem of frequent creation, writing, and deletion of small files on the shared storage device, greatly enhances the stability of the storage system, and since the first and second computing nodes in the cloud environment are located in the same high-speed network in the same availability zone, the transmission bandwidth is not a bottleneck. This solution makes full use of the abundant bandwidth resources.

[0114] Optional, in Figure 4 Based on the illustrated embodiment, the data synchronization process of the primary second computing node will be explained below. For example, Figure 5 This is an interactive schematic diagram of the data synchronization method provided in the second embodiment of this application. For example... Figure 5 As shown in the embodiments of this application, the copying of the first part of the data in the data segment to the second computing node via network transmission in step S402 above can be achieved through the following steps:

[0115] S501, the first computing node obtains a data snapshot of the first part of the data and the metadata corresponding to the data snapshot.

[0116] Optionally, in an embodiment of this application, when the read / write engine of the first computing node refreshes the data in the buffer into data segments, if it needs to synchronize the first part of the data to the second computing node, it will generate a data snapshot of the first part of the data. The data snapshot is a collection of segments corresponding to the first part of the data at the current moment.

[0117] In the embodiments of this application, during data synchronization, the first computing node can determine the metadata corresponding to the acquired data snapshot. This metadata is used to describe attribute information such as the length and position of the first part of the data.

[0118] It is understood that, in one optional embodiment of this application, after obtaining a data snapshot of the first portion of data, the first computing node may also perform the following steps:

[0119] S502, the first computing node adds the data segment from the data snapshot to the shared file list.

[0120] In one possible implementation, since the primary second computing node may need to synchronize multiple data segments during a single synchronization process, in order to perform data synchronization in an orderly manner, a shared file list is pre-established in the primary computing node. Thus, when the primary computing node generates a data segment, it can add all the data segments to be synchronized corresponding to that data snapshot to the shared file list.

[0121] Understandably, the shared file list is used to prevent files from being deleted by the first compute node while they are still being used by the second compute node. If the second compute node no longer needs a certain file, the first compute node can end the sharing and remove it from the shared file list.

[0122] S503, The first computing node sends the metadata corresponding to the data snapshot to the second computing node.

[0123] Optionally, after obtaining the metadata corresponding to the data snapshot, the first computing node can also send the metadata to the second computing node so that the second computing node can perform corresponding processing operations on the metadata and determine the data segment information that needs to be synchronized.

[0124] S504. After receiving the metadata, the second computing node sends a response message to the first computing node. The response message includes the loaded data snapshot information and the file to be shared in the second computing node.

[0125] In one possible implementation of this application embodiment, after receiving the metadata corresponding to the data snapshot, the second computing node can send a response message to the first computing node to indicate that the second computing node has received the metadata, or to indicate that the network link between the primary and second computing nodes is in a connected state.

[0126] In practical applications, the first computing node and the second computing node may need to perform multiple data synchronization processes. Therefore, the second computing node can determine the file to be shared (the file that needs to be shared) based on its currently loaded data segment information and received metadata. When the second computing node sends a response message to the first computing node, it can send its currently loaded data snapshot information and the file to be shared to the first computing node so that the first computing node can perform the corresponding processing.

[0127] Among them, the loaded data snapshot information can be understood as the version information of the data snapshot currently loaded by the second computing node, and the file to be shared is used to indicate the file information such as the data segment to be shared that the second computing node currently needs.

[0128] S505, the first computing node processes the shared file list based on the received loaded data snapshot information and the files to be shared, and determines the data segments in the shared file list that do not need to be shared.

[0129] S506. The first computing node removes the data segments that do not need to be shared from the shared file list.

[0130] In step S505, when the second computing node sends the loaded data snapshot information and the file to be shared back to the first computing node, the first computing node can receive the loaded data snapshot information and the file to be shared. Therefore, by processing the loaded data snapshot information, the version information of the data snapshot currently loaded by the second computing node can be determined. Then, combined with the file to be shared by the second computing node, the data segments that do not need to be shared in the shared file list can be determined.

[0131] Accordingly, in step S506, in order to avoid transmitting duplicate shared data segments between the primary and secondary computing nodes, the primary computing node can delete the unnecessary shared data segments from the shared file list after determining them, so as to improve transmission efficiency and avoid resource waste.

[0132] It is important to note that before the first compute node deletes a data segment that does not need to be shared from the shared file list, it needs to determine whether the shared file list is in use. If it is determined that the shared file list is in use, the deletion of the data segment in the shared file list is prohibited.

[0133] In other words, the fact that the shared file list is in use can be understood as the second compute node performing a data synchronization process. Therefore, in order to avoid deletion affecting the normal data segment synchronization process, it is necessary to ensure that the shared files in the shared file list are not deleted during the entire physical replication process. At the same time, if the second compute node is still using the shared file list for read traffic, the data segments in the shared file list cannot be deleted.

[0134] S507. The second computing node processes the received metadata and determines the target data segment to be copied.

[0135] In this embodiment, the second computing node can process the metadata to determine the version information and the data segment to be synchronized, and then combine it with the loaded data snapshot information in itself to determine the target data segment to be copied from the first computing node to the second computing node.

[0136] In one possible implementation, step S507, namely, the second computing node processes the received metadata to determine the target replicated data segment, may include: the second computing node processes the received metadata to determine the length information and index information of the data segment corresponding to the metadata, and then, based on the length information and index information of the data segment corresponding to the metadata, determines the target replicated data segment in the data segment corresponding to the metadata.

[0137] For example, metadata may include the length data and index information of the data segment to be synchronized. Accordingly, in this embodiment, the second computing node analyzes the received metadata to determine the length information and index information of the data segment to be processed. Then, it queries the loaded data snapshot information to determine whether the second computing node has already loaded the length information and index information of some data segments to be processed. If so, it filters out the data segments that have not been loaded in the second computing node and determines these data segments as target replication data segments.

[0138] Understandably, the second compute node calculates which data segments need to be copied by verifying metadata. It can also be interpreted that when the second compute node determines that the data segment to be synchronized is located in the memory of the first compute node (e.g., NRT cache), it can pull the data segment to be synchronized through network transmission. When it is determined that the data segment to be synchronized has been persisted to the shared storage device by the first compute node, the second compute node can directly load it from the shared storage device without needing to synchronize it through the first compute node.

[0139] S508, the second computing node sends a data replication request to the first computing node based on the target data segment to be replicated.

[0140] S509. Based on the received data replication request, the first computing node replicates the target data segment to the second computing node through physical replication.

[0141] Optionally, when the second computing node determines the target data segment to be synchronized, it can determine the target number of requests based on the number and length of each data segment in the target data segment, the size of each transmission, etc., and then send data replication requests to the first computing node cyclically based on the target number of requests until the target number of requests has been sent.

[0142] It is understandable that the second computing node cyclically sends data replication requests to the first computing node, which can be interpreted as the second computing node cyclically requesting incremental pull of the target data segment to be synchronized. Correspondingly, after receiving each data replication request, the first computing node can determine the target data segment to be replicated and copy it to the second computing node through physical replication.

[0143] In one optional embodiment, the first computing node, based on the received data replication request, copies the target data segment to the second computing node via physical replication. This includes: the first computing node, based on the file index information in the data replication request, determines the target data segment to be transferred in the shared file list, and copies the target data segment to the second computing node via physical replication. That is, when the second computing node cyclically requests incrementally fetched data segments, the first computing node will correspondingly send the data of the incrementally fetched data segments to the second computing node cyclically.

[0144] Understandably, after the replication process is completed, the second compute node can load the replicated data segments and provide read services to the outside world.

[0145] In the embodiments of this application, a first computing node obtains a data snapshot of a first portion of data and the corresponding metadata, and sends the metadata to a second computing node. The second computing node processes the received metadata to determine the target data segment to be copied. Then, based on the target data segment, it sends a data copy request to the first computing node. Based on the received data copy request, the first computing node copies the target data segment corresponding to the file index information to the second computing node using physical copying. In this technical solution, when the data segment is generated, the first and second computing nodes can achieve data synchronization through physical copying, reducing the number of network hops during data synchronization and lowering master-slave data latency.

[0146] Optionally, in the embodiments of this application, before the first computing node writes the received primary copy into memory, it may also perform the following operation: the first computing node writes a log file of the shared storage device based on the received primary copy. The log file is used for abnormal data recovery during the data synchronization process. Correspondingly, the data synchronization method may also include: the shared storage device rolling back the target data in the log file. The target data is data whose data segment has been persisted to the shared storage device.

[0147] In practical applications, after receiving the data written by the user (i.e., the primary replica), the first computing node directly uses the data to write the log file (translog) of the shared storage device. After writing, it can return a write success response to the user, which improves the user's writing experience.

[0148] In this embodiment, the first computing node writes data to the log file of the shared storage device, which can ensure the reliability of the data writing. That is, if the first computing node fails before the data is persisted to the shared storage device, the lost data can be recovered through the log file.

[0149] It is understandable that during the process of the first compute node persisting the data segment to the shared storage device through flush, the data segment already persisted to the shared storage device is available for the first compute node and the second compute node to read and load. Therefore, the log files in the shared storage device can be rolled back to reduce the storage space occupied by the logs and avoid the problem of long reading time caused by the first compute node and the second compute node restoring data based on the log files.

[0150] Based on the above analysis, the data synchronization method provided in this application is applied to a storage system including a first computing node, a second computing node, and a shared storage device. This storage system provides a separation architecture for computing and storage. The method uses two synchronization links, the shared storage device and the network transmission, to achieve master-slave data synchronization, which improves data write performance and reduces the pressure on the data and metadata of the shared storage device. Compared with master-slave synchronization that relies solely on the shared storage device, the point-to-point network transmission reduces the number of network hops and lowers the master-slave data latency.

[0151] Figure 6 This is a schematic block diagram of a computing node provided in an embodiment of this application. The computing node 600 can be... Figure 3 The first computing node in the algorithm can also be the second computing node. For example... Figure 6 As shown, the computing node 600 may include a receiving module 610, a processing module 620, and a sending module 630.

[0152] In one embodiment of this application, if computing node 600 is the first computing node, then:

[0153] The receiving module 610 is used to receive the master copy written by the user;

[0154] The processing module 620 is used to write the primary copy into memory and periodically refresh the primary copy in memory into data segments;

[0155] The processing module 620 is also used to copy the first part of the data in the above data segment to the second computing node through the sending module 630 based on the network transmission method when the memory refresh process ends, and to persistently store the data segment in the memory to the shared storage device through the sending module 630 when the preset storage conditions are met. The shared storage device is used for the second computing node to read and load the second part of the data in the unstored data segment. The storage conditions include that the total size of the data segment in the memory is greater than the capacity threshold, and / or that the time interval since the last storage is greater than the interval threshold.

[0156] Optionally, the processing module 620 is specifically used to obtain a data snapshot of the first part of the data, and send the metadata corresponding to the data snapshot to the second computing node through the sending module 630, receive the data replication request sent by the second computing node through the receiving module 610, and copy the target data segment to the second computing node through physical replication based on the received data replication request.

[0157] Optionally, the processing module 620 is further configured to add the data segment in the data snapshot to the shared file list after obtaining the data snapshot of the first part of the data, and determine the target data segment to be copied in the shared file list based on the file index information in the received data copy request. Correspondingly, the sending module 630 is configured to copy the target data segment to the second computing node by physical copying.

[0158] Optionally, the receiving module 610 is further configured to receive a response message fed back by the second computing node after receiving the metadata, the response message including the loaded data snapshot information and the file to be shared in the second computing node;

[0159] The processing module 620 is further configured to process the shared file list based on the received loaded data snapshot information and the file to be shared, determine the data segments that do not need to be shared in the shared file list, and delete the data segments that do not need to be shared from the shared file list.

[0160] It is understood that the processing module 620 is also used to prohibit the deletion of data segments in the shared file list when it is determined that the shared file list is in use.

[0161] Optionally, the processing module 620 is further configured to periodically merge the data segments in memory after periodically refreshing the primary copy in memory into data segments.

[0162] Optionally, the memory includes a buffer and an NRT cache; the processing module 620 is used to write the received primary copy into the buffer; periodically refresh the primary copy in the buffer into data segments in the NRT cache; and, when a preset storage condition is met, store the data segments in the NRT cache to a shared storage device through the sending module 630.

[0163] Optionally, the processing module 620 is further configured to write a log file of the shared storage device based on the received primary copy before writing the received primary copy to memory. The log file is used for abnormal data recovery during the data synchronization process. The shared storage device is configured to roll over and reclaim target data in the log file. The target data is data whose data segment has been persisted to the shared storage device.

[0164] In the embodiments of this application, the specific implementation scheme and beneficial effects of the first computing node are described in the above embodiments, and will not be repeated here.

[0165] Optionally, in another embodiment of this application, if computing node 600 is a second computing node, then:

[0166] Receiver module 610 is used to receive a first portion of synchronized data from the first computing node;

[0167] Processing module 620 is used to verify and merge the first part of the data to obtain the target data segment;

[0168] The sending module 630 is used to read and load the second part of the data from a shared storage device when it is determined that the second part of the data is not stored. The shared storage device is used to store the primary replica obtained by the first computing node, and the first part of the data and the second part of the data are secondary replicas of the primary replica.

[0169] In the embodiments of this application, the specific implementation scheme and beneficial effects of the second computing node are described in the above embodiments, and will not be repeated here.

[0170] Optionally, in an embodiment of this application, a shared storage device is provided, which includes a receiving module, a processing module, and a sending module.

[0171] The receiving module is used to receive a data segment to be stored from the first computing node. The data segment is sent by the first computing node when a preset storage condition is met. The storage condition includes that the total size of the data segments in the memory is greater than a capacity threshold, and / or that the time interval since the last storage is greater than an interval threshold.

[0172] The processing module is used to store the data segment to the target location based on the identifier of the first computing node;

[0173] The receiving module is also used to receive a data read request from the second computing node, which is issued by the second computing node when it determines that the second part of the data is not stored.

[0174] The processing module is further configured to determine the second part of the data based on the data reading request, and transmit it to the second computing node through the sending module.

[0175] In the embodiments of this application, the specific implementation scheme and beneficial effects of the shared storage device are described in the above embodiments, and will not be repeated here.

[0176] Figure 7 This is a schematic block diagram of an electronic device provided in an embodiment of this application. In one embodiment, the electronic device 700 can be a first computing node; in another embodiment, the electronic device 700 can be a second computing node. In yet another embodiment, the electronic device 700 can be a shared storage device.

[0177] like Figure 7 As shown, the electronic device 700 may include at least one processor 710 for implementing the data synchronization function in the method provided in this application. See the detailed description in the method examples for further details, which will not be repeated here.

[0178] The electronic device 700 may further include a memory 720 for storing program instructions and / or data. The memory 720 is coupled to the processor 710. The coupling in this application is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, for information exchange between devices, units, or modules. The processor 710 may operate in conjunction with the memory 720. The processor 710 may execute program instructions stored in the memory 720. At least one of the at least one memories may be included in the processor.

[0179] The electronic device 700 may further include a communication interface 730 for communicating with other devices via a transmission medium, thereby enabling the devices in the electronic device 700 to communicate with other devices. The communication interface 730 may be, for example, a transceiver, interface, bus, circuit, or a device capable of transmitting and receiving functions. The processor 710 may utilize the communication interface 730 to transmit and receive data and / or information, and to implement the data synchronization method described in the above embodiments.

[0180] This application does not limit the specific connection medium between the processor 710, memory 720, and communication interface 730. This application... Figure 7 The processor 710, memory 720, and communication interface 730 are connected via bus 740. Bus 740 is... Figure 7 The connections between other components are shown in bold and are for illustrative purposes only, not as limiting information. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0181] In the embodiments of this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0182] According to the method provided in this application, this application also provides a computer-readable storage medium storing a computer program, which, when run on a computer, causes the computer to execute the technical solutions in the above embodiments.

[0183] According to the method provided in this application, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute the technical solutions in the above embodiments.

[0184] The technical solutions provided in this application can be implemented in whole or in part through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a terminal device, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means, such as coaxial cable, optical fiber, digital subscriber line (DSL), or wireless means (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium such as a digital video disc (DVD), or a semiconductor medium, etc.

[0185] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data synchronization method, characterized in that, Applied to a storage system, the storage system including a first computing node, a second computing node, and a shared storage device, the method includes: When the first computing node refreshes the acquired primary copy into a data segment in memory, it copies the first part of the data segment to the second computing node via network transmission. The second computing node processes and stores the received first portion of data; When the preset storage conditions are met, the first computing node stores the data segments in the memory to the shared storage device. The storage conditions include that the total size of the data segments in the memory is greater than the capacity threshold, and / or that the time interval since the last storage is greater than the interval threshold. When the second computing node determines that the second part of the data segment is not stored, it reads and loads the second part of the data from the shared storage device, wherein the first part of the data and the second part of the data are slave replicas of the primary replica; Before the first computing node copies the first portion of data in the acquired primary replica to the second computing node via network transmission when the first computing node refreshes the primary replica into a data segment in memory, the method further includes: The first computing node writes the received primary replica into memory and periodically refreshes the data in the memory into the data segment; the first computing node periodically merges the data segments in the memory.

2. The method as described in claim 1, characterized in that, The first computing node copies a first portion of the data in the data segment to the second computing node via network transmission, including: The first computing node retrieves a data snapshot of the first portion of data from the memory and sends the metadata corresponding to the data snapshot to the second computing node; The second computing node processes the received metadata to determine the target data segment to be copied; The second computing node sends a data replication request to the first computing node based on the target data segment to be replicated; Based on the received data replication request, the first computing node replicates the target data segment to the second computing node via physical replication.

3. The method as described in claim 2, characterized in that, The second computing node processes the received metadata to determine the target data segment to be replicated, including: The second computing node processes the received metadata to determine the length and index information of the data segment corresponding to the metadata; The second computing node determines the target replicated data segment in the data segment corresponding to the metadata based on the length information and index information of the data segment corresponding to the metadata.

4. The method as described in claim 2, characterized in that, After the first computing node obtains a data snapshot of the first portion of data from the memory, the method further includes: The first computing node adds the data segments from the data snapshot to the shared file list; Based on the received data replication request, the first computing node replicates the target data segment to the second computing node via physical replication, including: Based on the file index information in the data copy request, the first computing node determines the target data segment to be copied from the shared file list; The first computing node copies the target data segment to the second computing node via physical replication.

5. The method as described in claim 4, characterized in that, Before the second computing node processes the received metadata and determines the target replicated data segment, the method further includes: After receiving the metadata, the second computing node sends a response message back to the first computing node. The response message includes the loaded data snapshot information and the file to be shared in the second computing node. The first computing node processes the shared file list based on the received loaded data snapshot information and the files to be shared, and determines the data segments in the shared file list that do not need to be shared; The first computing node removes the data segment that does not need to be shared from the list of shared files.

6. The method as described in claim 4 or 5, characterized in that, The method further includes: When the first computing node determines that the shared file list is in use, it prohibits the deletion of data segments in the shared file list.

7. The method according to any one of claims 1 to 5, characterized in that, The memory includes a buffer and a near real-time NRT cache; The first computing node writes the received primary replica into memory and periodically refreshes the data in the memory into data segments, including: The first computing node writes the received primary replica into the buffer; The first computing node periodically refreshes the primary replica in the buffer into data segments and stores them in the NRT cache; When preset storage conditions are met, the first computing node stores the data segment in the memory to the shared storage device, including: When the preset storage conditions are met, the first computing node stores the data segment in the NRT cache to the shared storage device.

8. The method according to any one of claims 1 to 5, characterized in that, Before the first computing node writes the received primary replica into memory, the method further includes: The first computing node writes a log file to the shared storage device based on the received primary replica. The log file is used for abnormal data recovery during the data synchronization process. The method further includes: The shared storage device recycles target data from the log files on a rolling basis. The target data is data whose data segment has been persisted to the shared storage device.

9. A storage system, characterized in that, The storage system includes a first computing node, a second computing node, and a shared storage device, wherein: The first computing node is used to copy a first part of the data in the acquired primary copy to the second computing node via network transmission when refreshing the acquired primary copy into a data segment in memory; The second computing node is used to process and store the received first part of the data; When the preset storage conditions are met, the first computing node is also used to store the data segments in the memory to the shared storage device. The storage conditions include that the total size of the data segments in the memory is greater than the capacity threshold, and / or that the time interval since the last storage is greater than the interval threshold. The second computing node is also configured to read and load the second portion of data from the shared storage device when it is determined that the second portion of data of the data segment is not stored, wherein the first portion of data and the second portion of data are slave replicas of the primary replica; Before the first computing node copies the first portion of data in the data segment to the second computing node via network transmission when it refreshes the acquired primary replica into a data segment in memory, the first computing node is further configured to: write the received primary replica into memory, and periodically refresh the data in memory into the data segment; and periodically merge the data segments in memory.

10. A data synchronization method, applied to a first computing node, characterized in that, The method includes: When the acquired primary copy is refreshed into a data segment in memory, the first part of the data segment is copied to the second computing node via network transmission. When the preset storage conditions are met, the data segments in the memory are persistently stored in the shared storage device. The shared storage device is used for the second computing node to read and load the second part of the data in the unstored data segments. The storage conditions include the total size of the data segments in the memory being greater than the capacity threshold, and / or the time interval from the previous storage being greater than the interval threshold. Before copying the first portion of data in the acquired primary replica to the second compute node via network transmission, the method further includes: The received primary copy is written into memory, and the data in the memory is periodically refreshed into the data segment; the data segments in the memory are periodically merged.

11. A data synchronization method applied to a second computing node, characterized in that, The method includes: The first compute node receives synchronized first part of data; the first part of data is copied from the first compute node to the second compute node via network transmission when the first compute node refreshes the acquired master copy into a data segment in memory; wherein, before the first compute node copies the first part of the data segment to the second compute node via network transmission when the first compute node refreshes the acquired master copy into a data segment in memory, the first compute node also writes the received master copy into memory, and periodically refreshes the data in memory into the data segment, and periodically merges the data segments in memory; The first part of the data is verified and merged to obtain the target data segment; If it is determined that the second part of the data is not stored, the second part of the data is read from and loaded from the shared storage device, which is used to store the primary copy obtained by the first computing node, and the first part of the data and the second part of the data are secondary copies of the primary copy.

12. An electronic device, characterized in that, Includes a processor for invoking a computer program to cause the electronic device to perform the method as described in any one of claims 1 to 8 and 10 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 8 and 10 to 11.

Images