Data synchronization method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data synchronization method and device, electronic equipment, a storage medium and a program product, and is applied to the field of distributed technologies. The data synchronization method is applied to a synchronization component in the data synchronization system, the system further comprises at least two data centers adopting key-value pairs to store data, and the specific method comprises the steps that data writing requests received by all the data centers are obtained; identifying requests for writing different values for the same target key request from the data writing requests to obtain conflict requests; identifying priorities of the conflict requests according to the additional features of the conflict requests, and determining a value carried by the conflict request with the highest priority as a target value; and generating a synchronization decision result according to the target key and the target value, and feeding back the synchronization decision result to each data center, so that each data center synchronizes the value of the target key into the target value. According to the embodiment of the invention, agency and message queue components in the traditional remote multi-activity architecture are eliminated, the system structure is simplified, and the system performance is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of distributed technology, and in particular, to a data synchronization method, device, electronic device, storage medium, and program product. Background Art

[0002] Multi-site active-active is a distributed system architecture that can maintain high availability of the system. This architecture establishes independent data centers in different geographical locations (such as different cities). These data centers need to process traffic and provide business services in daily operations. Because multiple data centers provide services at the same time, when different data centers modify the same data items, conflicts will occur, resulting in different data stored in each data center.

[0003] In order to keep the data stored in each data center consistent, the current practice is to set up a proxy in each data center. The write requests of each data center are written to a unified message queue through the local proxy, and then the proxy of each data center consumes the messages in the message queue to update the data in the local database.

[0004] This method requires the introduction of proxies and message queues, and the message queues are stateful, which increases the complexity of deployment and operation and maintenance; write requests need to be added to the message queue first and then consumed, which increases the processing delay of write requests. Summary of the invention

[0005] The embodiments of the present invention provide a data synchronization method, device, electronic device, storage medium and program product, which successfully eliminate the proxy and message queue components in the traditional multi-site active-active architecture, simplify the system structure and improve the system performance.

[0006] In a first aspect, a data synchronization method provided by an embodiment of the present invention is applied to a synchronization component in a data synchronization system, wherein the data synchronization system further includes at least two data centers, each data center using a key-value pair to store data, and the data synchronization method includes:

[0007] Obtain the write data requests initiated by the corresponding clients and received by each data center;

[0008] Identify requests for writing different values ​​for the same target key from data writing requests, and obtain conflicting requests;

[0009] identifying the priority of the conflicting request according to the additional features of the conflicting request, and determining the value carried by the conflicting request with the highest priority as the target value;

[0010] A synchronization decision result is generated according to the target key and the target value, and the synchronization decision result is fed back to each data center so that each data center synchronizes the value of the target key to the target value.

[0011] In a second aspect, a data synchronization device provided by an embodiment of the present invention is applied to a synchronization component in a data synchronization system, wherein the data synchronization system further includes at least two data centers, each data center uses a key-value pair to store data, and the data synchronization device includes:

[0012] An acquisition module is used to acquire write data requests initiated by corresponding clients and received by each data center;

[0013] An identification module, used for identifying requests for writing different values ​​for the same target key from write data requests, and obtaining conflicting requests;

[0014] A determination module, configured to identify the priority of the conflicting request according to the additional features of the conflicting request, and determine the value carried by the conflicting request with the highest priority as the target value;

[0015] The generation module is used to generate a synchronization decision result according to the target key and the target value, and feed back the synchronization decision result to each data center so that each data center synchronizes the value of the target key to the target value.

[0016] In a third aspect, an electronic device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, a data synchronization method as in any embodiment of the present invention is implemented.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, a data synchronization method as in any embodiment of the present invention is implemented.

[0018] In a fifth aspect, a computer program product provided by an embodiment of the present invention includes a computer program, and when the computer program is executed by a processor, the data synchronization method as described in any embodiment of the present invention is implemented.

[0019] In an embodiment of the present invention, a data center uses a key-value pair to store data, that is, the data center is implemented using a storage node that uses a key-value pair to store data, and a synchronization component is introduced. The synchronization component and the data center cooperate to directly process synchronization data requests, eliminating the links of the agent and the message queue in the traditional method. The synchronization component directly interacts with the data center and does not require an intermediate message queue for data transmission, which greatly simplifies the system architecture and reduces the complexity of deployment and operation and maintenance; the synchronization component directly processes write requests and performs data merging and synchronization, reduces intermediate operations, reduces the delay of write requests, and makes the system respond faster; the storage nodes that use key-value pairs to store data are functionally expanded, so as to cooperate with the synchronization component to achieve data synchronization, which reduces the difficulty of implementation; the synchronization component is responsible for conflict detection and data merging, and updates the merged data to each data center to ensure that the data in different data centers are eventually consistent; the synchronization component can perform data synchronization on a periodic basis, and is designed as a stateless component to support fast restart and replacement; each data center has a complete data set, provides high reliability and regional security, and has disaster recovery capabilities. That is, the embodiment of the present invention designs the data center as a storage node that uses key-value pairs to store data. On this basis, the synchronization component is introduced, which successfully eliminates the proxy and message queue components in the traditional multi-site active-active architecture, simplifies the system structure and improves the system performance; the stateless synchronization component design and the functional expansion of the storage node ensure data consistency, high availability and low-latency response, greatly improving the maintainability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a flow chart of a data synchronization method provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of a data synchronization system provided by an embodiment of the present invention;

[0023] Figure 3a is an example diagram of a process flow of a storage node processing data provided by an embodiment of the present invention;

[0024] Figure 3b is an example diagram of a process flow of a synchronization component processing data provided by an embodiment of the present invention;

[0025] Figure 3c is an example diagram of an offset application provided by an embodiment of the present invention;

[0026] Figure 4 is a structural diagram of a data synchronization device provided by an embodiment of the present invention;

[0027] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Figure 1 It is a flow chart of the data synchronization method provided by an embodiment of the present invention. The data synchronization method provided by an embodiment of the present invention can be applicable to the scenario of data synchronization between multiple data centers deployed in a distributed manner. The data synchronization method can be executed by the data synchronization device provided by an embodiment of the present invention, and the device can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, and the electronic device can be a synchronization component in a data synchronization system. The data synchronization system also includes at least two data centers, and each data center uses key-value pairs to store data, that is, each data center is implemented by a storage node that uses key-value pairs (KV) to store data. The following embodiment is described by taking the integration of the data synchronization device in the synchronization component as an example. See. Figure 1 The data synchronization method of this embodiment may include the following steps:

[0031] Step 110: Acquire the write data request initiated by the corresponding client and received by each data center.

[0032] Data centers are distributed in different geographical locations. These centers jointly provide services and data storage. Clients are terminal devices or software applications that initiate requests to data centers. They can be user computing devices (such as laptops, smartphones) or programs (such as web browsers, interface call programs). Their function is to initiate access requests to data. Write data requests are requests sent by clients to data centers to write or update data stored in the data center. Write data requests usually specify the data items to be written and the corresponding new values.

[0033] In a distributed system architecture, each data center can receive and process client requests within its scope of responsibility. After receiving a request from a client, each data center can store the client request in a local file and process the data stored in itself according to the client request. The received requests may include read data requests and write data requests. The read data request is used to request to read the corresponding data, and the write data request is used to request to add, delete or modify the corresponding data. The synchronization component can periodically pull the write data requests received by each data center from the local files of each data center, and prepare to synchronously update the data from different data centers under the conflict resolution mechanism to maintain global consistency.

[0034] Step 120 , identifying requests for writing different values ​​for the same target key from the write data requests, and obtaining conflicting requests.

[0035] Since write data requests come from different data centers, conflicts will occur when multiple write data requests attempt to modify the same target key at the same time, but the values ​​in these requests are different. Conflicting requests refer to data requests that attempt to write different values ​​to the same target key.

[0036] In a multi-data center environment of a distributed system, when processing multiple write data requests from clients at the same time, you may encounter multiple write data requests trying to modify the same target key at the same time, but the written values ​​are different, which will cause data inconsistency. Therefore, you can compare all collected write data requests, identify those for the same target key, and check these requests for the same target key to identify those that attempt to write different values. These requests are the source of data conflicts and are called conflicting requests.

[0037] Identifying conflicting requests is for subsequent conflict resolution steps to ensure data consistency and overall system stability in a distributed environment.

[0038] Step 130 , identifying the priority of the conflicting request according to the additional features of the conflicting request, and determining the value carried by the conflicting request with the highest priority as the target value.

[0039] Additional features refer to features or information that are additionally carried to assist in judgment and processing, in addition to the basic main features (such as keys and values) in the process of processing data requests or making decisions. Specifically in this embodiment, additional features include but are not limited to timestamps and numerical identifiers of data centers. Timestamps are identifiers used to record the time when an event occurs, which here refers to the time when the write data request is generated. Timestamps are generally in milliseconds or seconds and reflect the actual order in which the requests occur. Each data center can have a numerical identifier in the system to identify or distinguish different data centers and for sorting.

[0040] Specifically, it can be determined whether the timestamps of the conflicting requests are the same. If the timestamps of the conflicting requests are not the same, the latest request is determined from the conflicting requests according to the timestamps of the conflicting requests. The latest request has the highest priority, and the value carried by the latest request is determined as the target value. The latest request has the latest timestamp. By determining and comparing the timestamps of the conflicting requests, it is ensured that the latest value takes precedence from the time dimension.

[0041] If the timestamps of conflicting requests are the same, sort the data centers according to their numerical identifiers; identify the priority of conflicting requests based on the sorting results, and the conflicting requests that are ranked first or last have the highest priority; and determine the value carried by the conflicting request with the highest priority as the target value. That is, when the timestamps are the same, the final data value is determined by the pre-defined data center priority, thereby eliminating the conflict dilemma caused by the lack of time difference.

[0042] Step 140, generating a synchronization decision result according to the target key and the target value, and feeding back the synchronization decision result to each data center, so that each data center synchronizes the value of the target key to the target value.

[0043] The target key is the identifier of the specific data item that needs to be synchronized. The target value is the final value that is written to the target key. The target value is the value determined by the decision rule after processing the conflicting requests. The synchronization decision result is the synchronization instruction or result including the target key and target value generated based on the analysis and decision of the conflicting requests, which is used to guide the synchronization operation of each data center.

[0044] When processing conflicting requests, the target key and target value are determined through additional features such as timestamps and data center identifiers. At this time, this information needs to be packaged into a synchronization decision result, which is an instruction that clearly indicates that the target key should be written to the target value. The generated synchronization decision result is passed back to each data center in the system so that these data centers can understand the decision result and perform synchronization operations based on the result. After receiving the synchronization decision result, each data center updates the value of the target key in its local storage to the target value according to the instruction. In this way, the corresponding data items of all data centers are consistent.

[0045] In the specific implementation, in addition to conflicting requests, there are also non-conflicting requests in the write data request. Non-conflicting requests are requests other than conflicting requests in the write data request. The synchronization decision result can also be generated based on the target key and target value as well as the key value carried in the non-conflicting request. Non-conflicting requests can be applied directly because they will not cause data conflicts with other requests. Non-conflicting requests also need to be synchronized between various data centers.

[0046] That is to say, the synchronization decision result includes not only the target key and target value, but also the key value carried in the non-conflicting request. It is necessary to combine the key value carried in the non-conflicting request to generate the overall synchronization decision result, which will contain all the data instructions that need to be synchronized. Distinguishing between conflicting requests and non-conflicting requests, and then generating and propagating the synchronization decision result to all data centers based on the determined target key and target value, as well as the key-value pairs of non-conflicting requests, this process ensures data consistency and system stability.

[0047] In this embodiment, the data center uses key-value pairs to store data, that is, the data center is implemented using storage nodes that use key-value pairs to store data, and a synchronization component is introduced. The synchronization component and the data center cooperate to directly process synchronization data requests, eliminating the agent and message queue links in the traditional method. The synchronization component directly interacts with the data center and does not require an intermediate message queue for data transmission, which greatly simplifies the system architecture and reduces the complexity of deployment and operation and maintenance; the synchronization component directly processes write requests and performs data merger and synchronization, which reduces intermediate operations, reduces the delay of write requests, and makes the system respond faster; the storage nodes that use key-value pairs to store data are functionally expanded to cooperate with the synchronization component to achieve data synchronization, reducing the difficulty of implementation; the synchronization component is responsible for conflict detection and data merger, and updates the merged data to each data center to ensure that the data in different data centers are eventually consistent; the synchronization component can perform data synchronization on a periodic basis and is designed as a stateless component to support fast restart and replacement; each data center has a complete data set, provides high reliability and regional security, and has the ability to cope with disaster recovery needs. That is, the embodiment of the present invention designs the data center as a storage node that uses key-value pairs to store data. On this basis, the synchronization component is introduced, which successfully eliminates the proxy and message queue components in the traditional multi-site active-active architecture, simplifies the system structure and improves the system performance; the stateless synchronization component design and the functional expansion of the storage node ensure data consistency, high availability and low-latency response, greatly improving the maintainability and flexibility of the system.

[0048] The following describes the framework of the data synchronization system provided by the embodiment of the present invention. The exemplary framework can be as follows: Figure 2 As shown, it can include synchronization components and multiple data centers: Center A, Center B, and Center C. Each data center can be implemented using a storage node that stores KV data. The storage nodes of each data center can include a primary storage node and a backup storage node to prevent primary and backup switching in the event of a failure. Each primary storage node has an extension module, which extends the functions of the original storage node and adds commands that were not originally supported.

[0049] Each master storage node receives requests from clients, which may include write data requests and read data requests. The corresponding extension module filters out write data requests and stores the filtered write data requests in local files for the synchronization component to pull and use. The synchronization component can periodically (for example, every second) pull write data requests from the local files of the three master storage nodes. Each time, the write data request pulled is the latest write data request received by the master storage node. Conflicting requests are identified from these write data requests. After making decisions on the conflicting requests based on the additional features, a synchronization decision result is generated, and the synchronization decision result is fed back to each master storage node. Each master storage node updates its own stored data based on the synchronization decision result.

[0050] This embodiment implements the synchronization function through module expansion based on the original functions of the storage node, without modifying the source code of the storage node. The modular design improves the flexibility and adaptability of the implementation. By integrating the data synchronization function into the expansion module and synchronization component of the storage node, the introduction of agents and message queues in traditional solutions is avoided, the system architecture is simplified, and the complexity of deployment and operation and maintenance is reduced. The master-slave switchable design of the storage node further improves the high availability and fault recovery capabilities of the system.

[0051] The following describes the process of data processing by the main storage node of each data center, such as Figure 3a As shown, the main storage node can save the received client requests in the buffer, and the requests saved in the buffer include read data requests and write data requests. Read data requests do not require additional processing, and write data requests can be filtered out of the buffer through command calls, and write data requests can be cached separately. In order to prevent data loss, write data requests can also be written to the log. Afterwards, the main storage node can update its own data according to the write data request. For example, the write data request received by the main storage node of center A is a request to write X to 1, then the main storage node of center A here can write X to 1.

[0052] The synchronization component can periodically pull write data requests from the cache of each main storage node, merge the conflicting requests in the write data requests, generate a synchronization decision result, and feed back the synchronization decision result to the main storage nodes of each data center. The main storage node of each data center updates its own stored data according to the received synchronization decision result. For example, the write data request received by the main storage node of center A is a request to write X as 1, the write data request received by the main storage node of center B is a request to write X as 2, and the write data request received by the main storage node of center C is a request to write X as 3. Through conflict decision, it is found that the main storage node of center C received the latest write data request, so the decision result is to write X as 3. After the synchronization decision result is sent to the main storage nodes of each center, the main storage nodes of each center will write X as 3.

[0053] The following describes the process of synchronous component processing data, such as Figure 3bAs shown, after confirming that the connection with each data center is normal, the synchronization component sends a data pull request to the main storage node of each data center. The data pull request can carry the result of the last synchronization decision to achieve request reuse. If this is the first pull, the result of the last synchronization decision carried in the data pull request is empty. Asynchronously wait for all main storage nodes to feedback the latest received write data request that has not been uploaded to the synchronization component, identify conflicting requests from the pulled write data requests, merge the conflicting requests according to the additional features of the conflicting requests, and store the merged results in the buffer so that the synchronization decision results can be sent to the main storage nodes of each data center when the data is pulled next time.

[0054] Furthermore, offsets can be used to track data flow and processing progress, specifically:

[0055] Each master storage node can maintain an offset for the write data request reported by itself, which indicates that the previous write data request has been applied locally and successfully applied to other master storage nodes. When the synchronization component crashes or the master storage node crashes, after the synchronization component reconnects with each master storage node, each master storage node reports the write data request after the offset maintained by itself to the synchronization node.

[0056] The synchronization component can also maintain an offset for the write data requests it processes, which is used to count the write data requests it processes. The offset can start from 0. After sending each batch of data, the synchronization component can calculate its own current synchronization offset and the current synchronization offset of each main storage node, send its own current synchronization offset to each main storage node, and determine the synchronization result based on the feedback from each main storage node. If the synchronization results are all successful, the current synchronization offset of the corresponding main storage node is sent to each main storage node to prompt each main storage node to delete the corresponding data. The synchronization component is stateless. If the synchronization component fails, just start another synchronization component.

[0057] Specifically, the synchronization component can calculate the total number of pending requests this time based on the number of conflicting requests and the number of non-conflicting requests in the write data request. The conflicting requests will be merged into 1, so the total number of pending requests this time = the number of non-conflicting requests + 1. The last synchronization offset of the synchronization component and the total number of pending requests this time are added to obtain the current synchronization offset of the synchronization component. The current synchronization offset of the synchronization component is sent to each data center so that each data center can feedback synchronization success confirmation information for the current synchronization offset of the synchronization component after completing the current data synchronization, and receive synchronization success confirmation information fed back by each data center.

[0058] Specifically, the synchronization component can also count the number of write data requests from each data center to obtain the current number of requests for each data center; add the last synchronization offset of each data center and the current number of requests to obtain the current synchronization offset of the corresponding data center; send the current synchronization offset of the corresponding data center to each data center, so that the corresponding data center deletes the write data requests whose positions are before the current synchronization offset to release more storage space.

[0059] The following examples illustrate specific applications of the offset designed in the embodiments of the present invention. Figure 3c As shown:

[0060] Assuming that this is the first time to pull data, the synchronization component pulls 3 write data requests from center A, 1 write data request from center B, and 2 write data requests from center C, that is, a total of 6 write data requests are pulled this time. There are 3 conflicting requests among the 6 write data requests. Assuming that the 3 conflicting requests are the request from center A to write X as 1, the request from center B to write X as 2, and the request from center C to write X as 3. The request from center C has the latest timestamp, so the decision result is to write X as 3, and the 3 conflicting requests are merged into 1. At this time, for the synchronization component, the number of pending requests this time becomes 3 (that is, 3 non-conflicting requests) + 1 (that is, 3 conflicting requests are merged into 1), that is, 4. For the first synchronization, the last synchronization offset of the synchronization component is 0, so the current synchronization offset of the synchronization component is 4. Each center also reports a write data request for the first time, that is, the last synchronization offset of each center is 0. The synchronization offset of center A recorded by the synchronization component is 3 (3 write data requests came this time), the synchronization offset of center B is 1 (1 write data request came this time), and the synchronization offset of center C is 2 (2 write data requests came this time).

[0061] The synchronization component writes X as 3 and the key values ​​of the three non-conflicting requests in the synchronization decision result and feeds it back to the main storage nodes of each center. The main storage nodes of each center write X as 3 based on the synchronization decision result and update the corresponding data according to the key values ​​of the non-conflicting requests.

[0062] The synchronization component sends the current synchronization offset 4 of the synchronization component to each center. After completing the current data synchronization update, the main storage node of each center feeds back synchronization success confirmation information (Acknowledge character, ACK) for the current synchronization offset 4 of the synchronization component, and the message carries ACK=4.

[0063] After receiving the synchronization success confirmation information fed back by the main storage nodes of all data centers, the synchronization component feeds back its offset 3 to the main storage node of center A, feeds back its offset 1 to the main storage node of center B, and feeds back its offset 2 to the main storage node of center C. The main storage node of center A deletes the write data requests before offset 3, the main storage node of center B deletes the write data requests before offset 1, and the main storage node of center C deletes the write data requests before offset 2.

[0064] Figure 4 is a schematic diagram of a structure of a data synchronization device provided in an embodiment of the present invention, and the device is suitable for executing a data synchronization method provided in an embodiment of the present invention, such as Figure 4 As shown, the device is applied to a synchronization component in a data synchronization system, the data synchronization system also includes at least two data centers, each data center uses a key-value pair to store data, and the data synchronization device includes:

[0065] The acquisition module 401 is used to acquire the write data request initiated by the corresponding client and received by each data center;

[0066] An identification module 402 is used to identify requests for writing different values ​​for the same target key from write data requests, and obtain conflicting requests;

[0067] A determination module 403, configured to identify the priority of the conflicting request according to the additional features of the conflicting request, and determine the value carried by the conflicting request with the highest priority as the target value;

[0068] The generation module 404 is used to generate a synchronization decision result according to the target key and the target value, and feed back the synchronization decision result to each data center so that each data center synchronizes the value of the target key to the target value.

[0069] In one embodiment, the additional feature includes a timestamp, and the determination module 403 identifies the priority of the conflicting request according to the additional feature of the conflicting request, including:

[0070] Determine whether the timestamps of conflicting requests are the same;

[0071] If the timestamps of the conflicting requests are different, the latest request is determined from the conflicting requests according to the timestamps of the conflicting requests, and the latest request has the highest priority.

[0072] In one embodiment, the additional feature further includes a numerical identifier of the data center. After determining whether the timestamps of the conflicting requests are the same, the determination module 403 is further configured to:

[0073] If the timestamps of conflicting requests are the same, the data centers are sorted according to their numerical identifiers;

[0074] The priority of the conflicting requests is identified according to the sorting result, and the conflicting request sorted first or last has the highest priority.

[0075] In one embodiment, before generating the synchronization decision result according to the target key and the target value, the generation module 404 is further configured to:

[0076] Identify non-conflicting requests from the write data requests, where the non-conflicting requests are requests other than the conflicting requests from the write data requests;

[0077] Generates synchronization decision results based on the target key and target value and the key values ​​carried in the non-conflicting requests.

[0078] In one embodiment, the device further includes an offset application module, and the offset application module is used to:

[0079] The total number of pending requests is calculated based on the number of conflicting requests and the number of non-conflicting requests in the write data request;

[0080] The last synchronization offset of the synchronization component and the total number of pending requests are accumulated to obtain the current synchronization offset of the synchronization component.

[0081] Send the current synchronization offset of the synchronization component to each data center, so that each data center feeds back synchronization success confirmation information for the current synchronization offset of the synchronization component after completing the current data synchronization;

[0082] Receive synchronization success confirmation information fed back by each data center.

[0083] In one embodiment, the offset application module is further used to:

[0084] Count the number of write data requests from each data center this time to obtain the number of requests from each data center this time;

[0085] The last synchronization offset of each data center and the number of requests this time are accumulated to obtain the current synchronization offset of the corresponding data center;

[0086] The current synchronization offset of the corresponding data center is sent to each data center, so that the corresponding data center deletes the write data request whose position is before the current synchronization offset.

[0087] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0088] In an embodiment of the present invention, a data center uses a key-value pair to store data, that is, the data center is implemented using a storage node that uses a key-value pair to store data, and a synchronization component is introduced. The synchronization component and the data center cooperate to directly process synchronization data requests, eliminating the agent and message queue links in the traditional method. The synchronization component directly interacts with the data center and does not require an intermediate message queue for data transmission, which greatly simplifies the system architecture and reduces the complexity of deployment and operation and maintenance; the synchronization component directly processes write requests and performs data merging and synchronization, reduces intermediate operations, reduces the delay of write requests, and makes the system respond faster; the storage nodes that use key-value pairs to store data are functionally expanded to cooperate with the synchronization component to achieve data synchronization, reducing the difficulty of implementation; the synchronization component is responsible for conflict detection and data merging, and updates the merged data to each data center to ensure that the data in different data centers are eventually consistent; the synchronization component can perform data synchronization on a periodic basis, and is designed as a stateless component to support fast restart and replacement; each data center has a complete data set, provides high reliability and regional security, and has the ability to cope with disaster recovery needs. That is, the embodiment of the present invention designs the data center as a storage node that uses key-value pairs to store data. On this basis, the synchronization component is introduced, which successfully eliminates the proxy and message queue components in the traditional multi-site active-active architecture, simplifies the system structure and improves the system performance; the stateless synchronization component design and the functional expansion of the storage node ensure data consistency, high availability and low-latency response, greatly improving the maintainability and flexibility of the system.

[0089] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of an electronic device suitable for implementing an embodiment of the present invention. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0090] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the computer system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0091] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0092] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0093] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be arranged in a processor, for example, it may be described as follows: a processor includes an acquisition module, an identification module, a determination module, and a generation module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0096] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0097] Obtain write data requests initiated by the corresponding clients and received by each data center; identify requests for writing different values ​​for the same target key from the write data requests to obtain conflicting requests; identify the priority of the conflicting requests based on the additional features of the conflicting requests, and determine the value carried by the conflicting request with the highest priority as the target value; generate a synchronization decision result based on the target key and the target value, and feed back the synchronization decision result to each data center so that each data center synchronizes the value of the target key to the target value.

[0098] According to the technical solution of the embodiment of the present invention, the data center adopts key-value pairs to store data, that is, the data center is implemented by using storage nodes that adopt key-value pairs to store data, and a synchronization component is introduced. The synchronization component and the data center cooperate to directly process synchronization data requests, eliminating the links of agents and message queues in traditional methods. The synchronization component directly interacts with the data center and does not require an intermediate message queue for data transmission, which greatly simplifies the system architecture and reduces the complexity of deployment and operation and maintenance; the synchronization component directly processes write requests and performs data merging and synchronization, reduces intermediate operations, reduces the delay of write requests, and makes the system respond faster; the storage nodes that adopt key-value pairs to store data are functionally expanded, so as to cooperate with the synchronization component to realize data synchronization, which reduces the difficulty of implementation; the synchronization component is responsible for conflict detection and data merging, and updates the merged data to each data center to ensure that the data in different data centers are eventually consistent; the synchronization component can perform data synchronization on a periodic basis, and is designed as a stateless component to support fast restart and replacement; each data center has a complete data set, provides high reliability and regional security, and has the ability to cope with disaster recovery needs. That is, the embodiment of the present invention designs the data center as a storage node that uses key-value pairs to store data. On this basis, the synchronization component is introduced, which successfully eliminates the proxy and message queue components in the traditional multi-site active-active architecture, simplifies the system structure and improves the system performance; the stateless synchronization component design and the functional expansion of the storage node ensure data consistency, high availability and low-latency response, greatly improving the maintainability and flexibility of the system.

[0099] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data synchronization method provided in any embodiment of the present application.

[0100] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0102] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations, and the user needs to be informed and the user's consent or authorization is obtained. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted. In addition, the corresponding operation entrance is provided for the user to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0103] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that: A synchronization component applied to a data synchronization system, wherein the data synchronization system further includes at least two data centers, each data center uses a key-value pair to store data, and the data synchronization method includes: Obtain the write data requests initiated by the corresponding clients and received by each data center; Identify requests for writing different values ​​for the same target key from data writing requests, and obtain conflicting requests; identifying the priority of the conflicting request according to the additional features of the conflicting request, and determining the value carried by the conflicting request with the highest priority as the target value; A synchronization decision result is generated according to the target key and the target value, and the synchronization decision result is fed back to each data center so that each data center synchronizes the value of the target key to the target value.

2. The method according to claim 1, characterized in that Additional features include a timestamp, and the priority of conflicting requests is identified based on additional features of the conflicting requests, including: Determine whether the timestamps of conflicting requests are the same; If the timestamps of the conflicting requests are different, the latest request is determined from the conflicting requests according to the timestamps of the conflicting requests, and the latest request has the highest priority.

3. The method according to claim 2, characterized in that Additional features include a numerical identifier of the data center and, after determining whether the timestamps of the conflicting requests are identical, also include: If the timestamps of conflicting requests are the same, the data centers are sorted according to their numerical identifiers; The priority of the conflicting requests is identified according to the sorting result, and the conflicting request sorted first or last has the highest priority.

4. The method according to claim 1, characterized in that: Before generating a synchronous decision result based on the target key and target value, it also includes: Identify non-conflicting requests from the write data requests, where the non-conflicting requests are requests other than the conflicting requests from the write data requests; Generates synchronous decision results based on target keys and target values, including: Generates synchronization decision results based on the target key and target value and the key values ​​carried in the non-conflicting requests.

5. The method according to claim 4, characterized in that Also includes: The total number of pending requests is calculated based on the number of conflicting requests and the number of non-conflicting requests in the write data request; The last synchronization offset of the synchronization component and the total number of pending requests are accumulated to obtain the current synchronization offset of the synchronization component. Send the current synchronization offset of the synchronization component to each data center, so that each data center feeds back synchronization success confirmation information for the current synchronization offset of the synchronization component after completing the current data synchronization; Receive synchronization success confirmation information fed back by each data center.

6. The method according to claim 5, characterized in that Also includes: Count the number of write data requests from each data center this time to obtain the number of requests from each data center this time; The last synchronization offset of each data center and the number of requests this time are accumulated to obtain the current synchronization offset of the corresponding data center; The current synchronization offset of the corresponding data center is sent to each data center, so that the corresponding data center deletes the write data request whose position is before the current synchronization offset.

7. A data synchronization device, characterized in that: A synchronization component used in a data synchronization system, wherein the data synchronization system further includes at least two data centers, each data center using a key-value pair to store data, and the data synchronization device includes: An acquisition module is used to acquire write data requests initiated by corresponding clients and received by each data center; An identification module, used for identifying requests for writing different values ​​for the same target key from write data requests, and obtaining conflicting requests; A determination module, configured to identify the priority of the conflicting request according to the additional features of the conflicting request, and determine the value carried by the conflicting request with the highest priority as the target value; The generation module is used to generate a synchronization decision result according to the target key and the target value, and feed back the synchronization decision result to each data center so that each data center synchronizes the value of the target key to the target value.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the data synchronization method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data synchronization method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the data synchronization method according to any one of claims 1 to 6.

Images