Data synchronization method and device and distributed data processing system

Abstract

A data synchronization method and apparatus, and a distributed data processing system are applied to a first server of a first machine room, the first server comprises a first synchronization component, and the first synchronization component is in communication connection with a second synchronization component of a second server of a second machine room; the method comprises the following steps: acquiring first business data written into a first cache of a first machine room by a first client of the first machine room; synchronizing the first service data to the second synchronization component through the first synchronization component; wherein the first business data synchronized to the second synchronization component can be written into a second cache of the second machine room through the second server side, and the first cache and the second cache are in data isolation.

Description

Technical Field

[0001] The present disclosure relates to the field of cache technology, and in particular to a data synchronization method and device, and a distributed data processing system. Background Art

[0002] In some distributed systems, caches in different computer rooms cannot communicate with each other, so the cache framework can only be deployed in a single computer room. In the event of a power outage or natural disaster, a problem in one computer room will cause user data loss, making the service availability low. Summary of the invention

[0003] In a first aspect, an embodiment of the present disclosure provides a data synchronization method, which is applied to a first server in a first computer room, wherein the first server includes a first synchronization component, and the first synchronization component is communicatively connected to a second synchronization component of a second server in a second computer room; the method includes: obtaining first business data written by a first client in the first computer room to a first cache of the first computer room; synchronizing the first business data to the second synchronization component through the first synchronization component; wherein the first business data synchronized to the second synchronization component can be written to the second cache of the second computer room through the second server, and data is isolated between the first cache and the second cache.

[0004] In some embodiments, the first server and the second server are both first type servers, the first server further includes a first service component, and the second server further includes a second service component; the obtaining of the first business data written by the first client of the first computer room into the first cache of the first computer room includes: obtaining the first business data written by the first client into the first cache through the first service component; writing the first business data obtained by the first service component into the first synchronization component; wherein the first business data synchronized to the second synchronization component can be obtained by the second service component and written into the second cache by the second service component.

[0005] In some embodiments, writing the first business data acquired by the first service component into the first synchronization component includes: determining the target tenant to which the first business data belongs; determining the target synchronization link corresponding to the target tenant, different synchronization links being data transmission links between different first computer rooms and / or different second computer rooms; writing the first business data into the target message queue corresponding to the target synchronization link in the first synchronization component; wherein the first synchronization component includes at least one message queue, and each message queue in the at least one message queue corresponds to a synchronization link.

[0006] In some embodiments, the first client includes a configuration file, and the configuration file includes a calling address; if the calling address is empty, the first server is a server of the first type.

[0007] In some embodiments, the first server and the second server are both second-type servers, and the first server also includes a first service component; obtaining the first business data written by the first client of the first computer room to the first cache of the first computer room includes: monitoring the operation of the first client writing the first business data to the first cache through the first service component; if monitored, obtaining the first business data from the first cache through the first synchronization component; wherein the first business data synchronized to the second synchronization component can be written into the second cache by the second synchronization component.

[0008] In some embodiments, the number of the first server terminals is greater than 1, and the first business data carries service component identification information for identifying one of the multiple first server terminals; the obtaining of the first business data written by the first client of the first computer room into the first cache of the first computer room includes: determining whether the service component identification information corresponding to the first server terminal is the same as the service component identification information carried in the first business data; wherein different first service components correspond to different service component identification information; and if so, obtaining the first business data written by the first client of the first computer room into the first cache of the first computer room.

[0009] In some embodiments, obtaining the first business data written by the first client of the first computer room to the first cache of the first computer room includes: if the service component identification information corresponding to the first server changes, before the change is completed, writing the first business data written to the first cache into a blocking queue; after the change is completed, obtaining the first business data from the blocking queue.

[0010] In some embodiments, the first client includes a configuration file, and the configuration file includes a calling address; if the calling address is not empty, the first server is the second type of server.

[0011] In some embodiments, the first business data carries the computer room identifier of the computer room that sends the first business data; wherein, if the computer room identifier carried in the first business data is the same as the computer room identifier of the second computer room, the first business data is not written into the second cache; and / or if the computer room identifier carried in the first business data is different from the computer room identifier of the second computer room, the first business data is written into the second cache.

[0012] In some embodiments, the method also includes: synchronizing the first vector clock and the first system clock corresponding to the first business data to the second synchronization component through the first synchronization component; wherein the first vector clock and the first system clock synchronized to the second synchronization component are used by the second server to determine whether to write the first business data synchronized to the second synchronization component into the second cache.

[0013] In some embodiments, if any of the following conditions is met, it is determined that the first business data synchronized to the second synchronization component is written into the second cache: the value of the first vector clock is greater than the value of the second vector clock in the second cache; the value of the first vector clock is equal to the value of the second vector clock, and the value of the first system clock is greater than the value of the second system clock in the second cache; the value of the first vector clock is equal to the value of the second vector clock, the value of the first system clock is equal to the value of the second system clock, and the priority of the first computer room is higher than the priority of the second computer room.

[0014] In some embodiments, the first vector clock and the first system clock are stored in the first cache in the form of a key-value pair, wherein the field name of the first vector clock serves as the key of the key-value pair, and the value of the first vector clock and the value of the first system clock are hashed as the value of the key-value pair, and each time business data is written to the first cache, the value of the first vector clock is updated according to a preset update method.

[0015] In some embodiments, the first computer room further includes a first verification component, and the second computer room further includes a second verification component, and the first verification component is communicatively connected to the second verification component; the second verification component can obtain the first business data synchronized to the second synchronization component, and synchronize the obtained first business data to the first verification component; the first verification component can obtain the first business data written to the first cache through the first synchronization component, and perform consistency verification on the first business data synchronized to the second synchronization component based on the first business data obtained from the second verification component and the first business data obtained from the first synchronization component.

[0016] In some embodiments, the second verification component subscribes to first business data from a message queue corresponding to the target synchronization link in the second synchronization component based on a target synchronization link between the second computer room and the first computer room, and sends the subscribed first business data to the first verification component in the first computer room; wherein different synchronization links are data transmission links between different first computer rooms and / or different second computer rooms, and each synchronization link corresponds to a message queue.

[0017] In some embodiments, the first business data obtained by the first verification component from the second verification component and the first business data obtained by the first verification component from the first synchronization component are both stored in the etcd of the first computer room, and the first verification component reads them from the etcd of the first computer room and performs consistency verification.

[0018] In some embodiments, the consistency check includes: performing consistency check on the key of the first business data obtained from the second verification component and the key of the first business data obtained from the first synchronization component; the value of the first business data stored in etcd of the first computer room is set to a preset value, and the storage space occupied by the preset value is smaller than the storage space occupied by the actual value of the first business data.

[0019] In some embodiments, the consistency check also includes: based on the source vector clock and the first system clock of the first computer room and the second vector clock and the second system clock of the second computer room, checking the value of the first business data obtained from the second verification component and the value of the first business data obtained from the first synchronization component.

[0020] In a second aspect, an embodiment of the present disclosure provides a data synchronization method, which is applied to a second server in a second computer room, wherein the second server includes a second synchronization component, and the second synchronization component is communicatively connected to the first synchronization component in the first computer room; the method includes: monitoring an operation of synchronizing first business data to the second synchronization component; after the first business data is written to a first cache of the first computer room by a first client in the first computer room, the first business data is acquired by the first server in the first computer room and synchronized to the second synchronization component through the first synchronization component; if monitored, the first business data synchronized to the second synchronization component is written to the second cache of the second computer room.

[0021] In some embodiments, the first business data carries a computer room identifier of a computer room that sends the first business data; and writing the first business data synchronized to the second synchronization component into the second cache of the second computer room includes: if the computer room identifier carried in the first business data is the same as the computer room identifier of the second computer room, not writing the first business data into the second cache; and / or if the computer room identifier carried in the first business data is different from the computer room identifier of the second computer room, writing the first business data into the second cache.

[0022] In some embodiments, the method also includes: monitoring a first vector clock and a first system clock synchronized to the second synchronization component, the first vector clock and the first system clock being the vector clock and the system clock corresponding to the first business data; obtaining the second vector clock and the second system clock in the second cache; and judging whether to write the first business data synchronized to the second synchronization component into the second cache based on the first vector clock, the first system clock, the second vector clock and the second system clock.

[0023] In some embodiments, the second vector clock and the second system clock are stored in the second cache in the form of a key-value pair, wherein the field name of the second vector clock serves as the key of the key-value pair, the value of the second vector clock and the value of the second system clock are hashed as the value of the key-value pair, and the value of the second vector clock is updated according to a preset update method.

[0024] In a third aspect, an embodiment of the present disclosure provides a data synchronization device, which is applied to a first server in a first computer room, wherein the first server includes a first synchronization component, and the first synchronization component is communicatively connected to a second synchronization component of a second server in a second computer room; the device includes: an acquisition module, for acquiring first business data written by a first client in the first computer room into a first cache of the first computer room; a synchronization module, for synchronizing the first business data to the second synchronization component through the first synchronization component; wherein the first business data synchronized to the second synchronization component can be written to the second cache of the second computer room through the second server, and data is isolated between the first cache and the second cache.

[0025] In some embodiments, the first server and the second server are both first type servers, the first server also includes a first service component, and the second server also includes a second service component; the acquisition module is specifically used to: acquire the first business data written to the first cache by the first client through the first service component; write the first business data acquired by the first service component into the first synchronization component; wherein the first business data synchronized to the second synchronization component can be acquired by the second service component and written into the second cache by the second service component.

[0026] In some embodiments, the acquisition module is specifically used to: determine the target tenant to which the first business data belongs; determine the target synchronization link corresponding to the target tenant, different synchronization links are data transmission links between different first computer rooms and / or different second computer rooms; write the first business data into the target message queue corresponding to the target synchronization link in the first synchronization component; wherein the first synchronization component includes at least one message queue, and each message queue in the at least one message queue corresponds to a synchronization link.

[0027] In some embodiments, the first client includes a configuration file, and the configuration file includes a calling address; if the calling address is empty, the first server is a server of the first type.

[0028] In some embodiments, the first server and the second server are both second-type servers, and the first server also includes a first service component; the acquisition module is specifically used to: monitor the operation of the first client writing first business data to the first cache through the first service component; if monitored, obtain the first business data from the first cache through the first synchronization component; wherein the first business data synchronized to the second synchronization component can be written into the second cache by the second synchronization component.

[0029] In some embodiments, the number of the first server terminals is greater than 1, and the first business data carries service component identification information for identifying one of the multiple first server terminals; the acquisition module is specifically used to: determine whether the service component identification information corresponding to the first server terminal is the same as the service component identification information carried in the first business data; wherein different first service components correspond to different service component identification information; if so, obtain the first business data written by the first client of the first computer room into the first cache of the first computer room.

[0030] In some embodiments, the acquisition module is specifically used to: if the service component identification information corresponding to the first server changes, before the change is completed, write the first business data written into the first cache into the blocking queue; after the change is completed, obtain the first business data from the blocking queue.

[0031] In some embodiments, the first client includes a configuration file, and the configuration file includes a calling address; if the calling address is not empty, the first server is the second type of server.

[0032] In some embodiments, the first business data carries the computer room identifier of the computer room that sends the first business data; wherein, if the computer room identifier carried in the first business data is the same as the computer room identifier of the second computer room, the first business data is not written into the second cache; and / or if the computer room identifier carried in the first business data is different from the computer room identifier of the second computer room, the first business data is written into the second cache.

[0033] In some embodiments, the device also includes: a synchronization module, used to synchronize the first vector clock and the first system clock corresponding to the first business data to the second synchronization component through the first synchronization component; wherein the first vector clock and the first system clock synchronized to the second synchronization component are used by the second server to determine whether to write the first business data synchronized to the second synchronization component into the second cache.

[0034] In some embodiments, if any of the following conditions is met, it is determined that the first business data synchronized to the second synchronization component is written into the second cache: the value of the first vector clock is greater than the value of the second vector clock in the second cache; the value of the first vector clock is equal to the value of the second vector clock, and the value of the first system clock is greater than the value of the second system clock in the second cache; the value of the first vector clock is equal to the value of the second vector clock, the value of the first system clock is equal to the value of the second system clock, and the priority of the first computer room is higher than the priority of the second computer room.

[0035] In some embodiments, the first vector clock and the first system clock are stored in the first cache in the form of a key-value pair, wherein the field name of the first vector clock serves as the key of the key-value pair, and the value of the first vector clock and the value of the first system clock are hashed as the value of the key-value pair, and each time business data is written to the first cache, the value of the first vector clock is updated according to a preset update method.

[0036] In some embodiments, the first computer room further includes a first verification component, and the second computer room further includes a second verification component, and the first verification component is communicatively connected to the second verification component; the second verification component can obtain the first business data synchronized to the second synchronization component, and synchronize the obtained first business data to the first verification component; the first verification component can obtain the first business data written to the first cache through the first synchronization component, and perform consistency verification on the first business data synchronized to the second synchronization component based on the first business data obtained from the second verification component and the first business data obtained from the first synchronization component.

[0037] In some embodiments, the second verification component subscribes to first business data from a message queue corresponding to the target synchronization link in the second synchronization component based on a target synchronization link between the second computer room and the first computer room, and sends the subscribed first business data to the first verification component in the first computer room; wherein different synchronization links are data transmission links between different first computer rooms and / or different second computer rooms, and each synchronization link corresponds to a message queue.

[0038] In some embodiments, the first business data obtained by the first verification component from the second verification component and the first business data obtained by the first verification component from the first synchronization component are both stored in the etcd of the first computer room, and the first verification component reads them from the etcd of the first computer room and performs consistency verification.

[0039] In some embodiments, the consistency check includes: performing consistency check on the key of the first business data obtained from the second verification component and the key of the first business data obtained from the first synchronization component; the value of the first business data stored in etcd of the first computer room is set to a preset value, and the storage space occupied by the preset value is smaller than the storage space occupied by the actual value of the first business data.

[0040] In some embodiments, the consistency check also includes: based on the source vector clock and the first system clock of the first computer room and the second vector clock and the second system clock of the second computer room, checking the value of the first business data obtained from the second verification component and the value of the first business data obtained from the first synchronization component.

[0041] In a fourth aspect, an embodiment of the present disclosure provides a data synchronization device, which is applied to a second server in a second computer room, wherein the second server includes a second synchronization component, and the second synchronization component is communicatively connected to the first synchronization component of the first computer room; the device includes: a monitoring module, which is used to monitor the operation of synchronizing first business data to the second synchronization component; after the first business data is written to the first cache of the first computer room by the first client of the first computer room, it is acquired by the first server in the first computer room and synchronized to the second synchronization component through the first synchronization component; a writing module, which is used to write the first business data synchronized to the second cache of the second computer room if monitored.

[0042] In some embodiments, the first business data carries a computer room identifier of a computer room that sends the first business data; the writing module is specifically used to: if the computer room identifier carried in the first business data is the same as the computer room identifier of the second computer room, not write the first business data into the second cache; and / or if the computer room identifier carried in the first business data is different from the computer room identifier of the second computer room, write the first business data into the second cache.

[0043] In some embodiments, the device also includes: a second monitoring module, used to monitor the first vector clock and the first system clock synchronized to the second synchronization component, the first vector clock and the first system clock are the vector clock and the system clock corresponding to the first business data; a clock acquisition module, used to obtain the second vector clock and the second system clock in the second cache; a judgment module, used to judge whether to write the first business data synchronized to the second synchronization component into the second cache based on the first vector clock, the first system clock, the second vector clock and the second system clock.

[0044] In some embodiments, the second vector clock and the second system clock are stored in the second cache in the form of a key-value pair, wherein the field name of the second vector clock serves as the key of the key-value pair, the value of the second vector clock and the value of the second system clock are hashed as the value of the key-value pair, and the value of the second vector clock is updated according to a preset update method.

[0045] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present disclosure.

[0046] In a sixth aspect, an embodiment of the present disclosure provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present disclosure when executing the program.

[0047] In the seventh aspect, an embodiment of the present disclosure provides a distributed data processing system, including a first server, a first client and a first cache located in a first computer room, and a second server and a second cache located in a second computer room; the first server includes a first synchronization component, the second server includes a second synchronization component, and the first synchronization component is communicatively connected to the second synchronization component; the first client is used to write first business data to the first cache; the first server is used to obtain the first business data written to the first cache, and synchronize the first business data to the second synchronization component through the first synchronization component; the second server is used to write the first business data synchronized to the second synchronization component into the second cache, and the data between the second cache and the first cache is isolated.

[0048] In the disclosed embodiment, the first computer room includes a first synchronization component, the second computer room includes a second synchronization component, and the first synchronization component is in communication connection with the second synchronization component. Therefore, the first business data written into the first cache of the first computer room by the first client of the first computer room can be synchronized to the second synchronization component through the first synchronization component, and then the first business data synchronized to the second synchronization component is written into the second cache of the second computer room by the second server of the second computer room, thereby synchronizing the first business data in the first cache to the second cache under the condition of data isolation between the first cache and the second cache.

[0049] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings herein are incorporated into the specification and constitute a part of the present disclosure. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0051] Figure 1 It is a flowchart of the data synchronization method of the embodiment of the present disclosure.

[0052] Figure 2 It is a schematic diagram of the system architecture in a cloud scenario of an embodiment of the present disclosure.

[0053] Figure 3 It is a schematic diagram of the system architecture in a non-cloud scenario of an embodiment of the present disclosure.

[0054] Figure 4 It is a schematic diagram of the data loop problem of an embodiment of the present disclosure.

[0055] Figure 5 It is a schematic diagram of a solution to duplicate consumption according to an embodiment of the present disclosure.

[0056] Figure 6It is a schematic diagram of the processing process when there are multiple replicators in an embodiment of the present disclosure.

[0057] Figure 7 It is a schematic diagram of the client initialization process of an embodiment of the present disclosure.

[0058] Figure 8 It is a schematic diagram of the process of calling the packaging class according to an embodiment of the present disclosure.

[0059] Fig. 9 It is a schematic diagram of a data conflict situation according to an embodiment of the present disclosure.

[0060] Fig.10 is a schematic diagram of a vector clock according to an embodiment of the present disclosure.

[0061] Fig.11 It is a schematic diagram of a situation where data synchronization fails according to an embodiment of the present disclosure.

[0062] Fig.12 It is a schematic diagram of an implementation framework of data consistency verification according to an embodiment of the present disclosure.

[0063] Fig.13 It is a schematic diagram of the data consistency verification process of an embodiment of the present disclosure.

[0064] Fig.14 It is a schematic diagram of the distribution process of synchronization requests according to an embodiment of the present disclosure.

[0065] Fig.15 It is a schematic diagram of a configuration method of a custom identification field according to an embodiment of the present disclosure.

[0066] Fig.16 It is a schematic diagram of a data synchronization method in a non-cloud scenario according to an embodiment of the present disclosure.

[0067] Fig.17 Schematic diagram of the processing logic of the vector clock according to an embodiment of the present disclosure.

[0068] Fig.18 is a flowchart of a data synchronization method according to another embodiment of the present disclosure.

[0069] Fig.19 is a schematic diagram of a data synchronization device according to an embodiment of the present disclosure.

[0070] Fig. 20 is a schematic diagram of a data synchronization device according to another embodiment of the present disclosure.

[0071] Fig.21 is a schematic diagram of a computer device according to an embodiment of the present disclosure.

[0072] Fig. 22 It is a distributed data processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0074] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0075] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0076] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0077] In some distributed systems, caches in different computer rooms cannot communicate with each other. Taking Redis cache as an example, Redis cache is a high-performance key-value cache system that supports storing multiple types of data such as strings, lists, sets, ordered sets (zsets) and hashes, and Redis is also a cache framework that is used frequently. In related technologies, due to the natural characteristics of Redis, Redis business data in different computer rooms cannot communicate with each other, and the Redis cache framework can only be deployed in a single computer room. Assuming that a user logs in in computer room A, the user's session can only be stored in the Redis cache of computer room A, and the Redis cache of other computer rooms cannot obtain the user's session. Therefore, all business requests of the user will be forwarded to the service of computer room A within the validity period of the session, and cannot be forwarded to other computer rooms, otherwise the user's business request will be kicked out because the user's session is not recorded in the Redis cache of other computer rooms. In the case of power outages in computer room A or natural disasters, the above solution is likely to cause user data loss, resulting in low service availability.

[0078] Based on this, the embodiment of the present disclosure provides a data synchronization method, which is applied to a first server in a first computer room, wherein the first server includes a first synchronization component, and the first synchronization component is communicatively connected to a second synchronization component of a second server in a second computer room. Figure 1 , the method comprising:

[0079] Step S11: obtaining first service data written by a first client in a first computer room into a first cache in the first computer room;

[0080] Step S12: synchronizing the first service data to the second synchronization component through the first synchronization component;

[0081] The first business data synchronized to the second synchronization component can be written into the second cache of the second computer room through the second server, and the data between the first cache and the second cache is isolated.

[0082] In some embodiments, the computer room can be divided into a source computer room (denoted as IDC_S) and a destination computer room (denoted as IDC_O) based on the transmission direction of the business data. Among them, if computer room A synchronizes the business data of this computer room to computer room B, computer room A is called the source computer room IDC_S, and computer room B is called the destination computer room IDC_O. It can be understood that the same computer room can be both the source computer room and the destination computer room. For example, computer room A can synchronize business data to computer room B, and at the same time, computer room A can also obtain business data synchronized to computer room A from computer room B or other computer rooms (for example, computer room C). When computer room A synchronizes business data to computer room B, computer room A can serve as the source computer room IDC_S, and computer room B can serve as the destination computer room IDC_O; and when computer room A obtains business data synchronized from computer room B or computer room C, computer room A can serve as the destination computer room IDC_O, and computer room B or computer room C can serve as the source computer room IDC_S.

[0083] In the embodiments of the present disclosure, the process of synchronizing business data from the first computer room to the second computer room is mainly described. Therefore, in the following, the first computer room is referred to as the source computer room IDC_S, and the second computer room is referred to as the destination computer room IDC_O. Correspondingly, the first server in the first computer room is referred to as the source server Server_S, the first synchronization component is referred to as the source synchronization component Sync_S, the first client is referred to as the source client App_S, and the first cache is referred to as the source cache Cache_S; similarly, the second server, the second synchronization component, the second client, and the second cache in the second computer room can be referred to as the destination server Server_O, the destination synchronization component Sync_O, the destination client App_O, and the destination cache Cache_O, respectively. In some embodiments, both the first cache and the second cache can be Redis caches. In this case, the first cache and the second cache can be respectively recorded as Redis_S and Redis_O. In the following embodiments, the first cache and the second cache are both Redis caches for illustration. It can be understood that this is not a limitation of the present disclosure.

[0084] The disclosed embodiment adopts an architecture in which the client and the server are separated, wherein, in the source computer room IDC_S, the first business data is written to the source Redis cache Redis_S through the source client App_S; and the source server Server_S communicates with the destination synchronization component Sync_O set by the destination server Server_O by setting the source synchronization component Sync_S, so that the first business data written to the source Redis cache Redis_S is synchronized to the destination synchronization component Sync_O through the source synchronization component Sync_S, and then the first business data synchronized to the destination synchronization component Sync_O is written to the destination Redis cache through the destination server Server_O. In the above manner, when the data is isolated between the Redis caches in different computer rooms, data synchronization between the Redis caches in different computer rooms is achieved. When the business data in one computer room is unavailable, services can still be provided through other computer rooms, thereby improving business availability.

[0085] The number of source computer room IDC_S and the number of destination computer room IDC_O can both be greater than or equal to 1. For example, when synchronizing the business data of computer room A to computer rooms B1 and B2, the number of source computer room IDC_S is equal to 1, and the number of destination computer room IDC_O is greater than 1; for another example, when synchronizing the business data of computer rooms B1 and B2 to computer room A, the number of source computer room IDC_S is greater than 1, and the number of destination computer room IDC_O is equal to 1; when synchronizing the business data of computer room A to computer rooms B1 and B2, and synchronizing the business data of computer room C to computer rooms D1 and D2, the number of source computer room IDC_S and the number of destination computer room IDC_O are both greater than 1.

[0086] The number of source clients App_S in the source computer room IDC_S can be equal to 1, that is, each source computer room IDC_S corresponds to one source client App_S. Similarly, the destination computer room IDC_O can include destination clients App_O, and each destination computer room IDC_O corresponds to one destination client App_O. The number of source servers Server_S in the source computer room IDC_S and the number of destination servers Server_O in the destination computer room IDC_O can both be greater than or equal to 1. The number of source Redis caches Redis_S in the source computer room IDC_S and the number of destination Redis caches in the destination computer room IDC_O can both be greater than or equal to 1. When the number of source Redis caches Redis_S in the source computer room IDC_S is greater than 1, one of the source Redis caches Redis_S can be used as the master node, and the other source Redis caches Redis_S can be used as slave nodes. Inside the source computer room IDC_S, the business data of the master node can be synchronized to the slave nodes of the source computer room IDC_S; when data synchronization is performed across computer rooms, the business data can be synchronized to the destination computer room IDC_O through the master node of the source computer room IDC_S. Similarly, when the number of destination Redis caches in the destination computer room IDC_O is greater than 1, one of the destination Redis caches can be used as the master node, and the other destination Redis caches can be used as slave nodes. The business data synchronized from the source computer room IDC_S can be obtained through the master node of the destination computer room IDC_O, and then the master node of the destination computer room IDC_O synchronizes the received business data to the slave nodes of the destination computer room IDC_O.

[0087] Each source server Server_S may include a source synchronization component Sync_S, and each destination server Server_O may include a destination synchronization component Sync_O. The source synchronization component Sync_S and the destination synchronization component Sync_O may be connected via a communication port, thereby achieving data synchronization between different computer rooms.

[0088] In some embodiments, the first service end includes a first service component and a first synchronization component, and the second service end includes a second service component and a second synchronization component. Continuing with the previous example, when the first computer room is the source computer room IDC_S, the first service component can be called the source service component, and the second service component can be called the destination service component. Optionally, the source service component can obtain the first business data written by the source client App_S to the source Redis cache Redis_S, and then write the obtained first business data to the source synchronization component Sync_S. Similarly, the first business data synchronized by the source synchronization component Sync_S to the destination synchronization component Sync_O can be obtained by the destination service component and written by the destination service component to the destination Redis cache Redis_O. That is, in this embodiment, in the source computer room, the first business data undergoes two transmission processes from the source Redis cache Redis_S to the source synchronization component Sync_S, one is transmitted from the source Redis cache Redis_S to the source service component, and the other is written from the source service component to the source synchronization component Sync_S. In the destination computer room, the first business data also undergoes two transmission processes from the destination synchronization component Sync_O to the destination Redis cache Redis_O, one from the synchronization component Sync_O to the destination service component, and the other from the destination service component to the destination Redis cache Redis_O.

[0089] Or, optionally, the source service component may only monitor the operation of the source client App_S writing the first business data to the source Redis cache Redis_S. If monitored, the first business data is obtained from the source Redis cache Redis_S through the source synchronization component Sync_S, and the source synchronization component Sync_S synchronizes the obtained first business data to the destination synchronization component Sync_O. The first business data synchronized by the source synchronization component Sync_S to the destination synchronization component Sync_O can be directly written to the destination Redis cache Redis_O by the destination synchronization component Sync_O. In this case, the first business data is only operated once from the source Redis cache Redis_S to the source synchronization component Sync_S, that is, the source synchronization component Sync_S obtains the first business data from the source Redis cache Redis_S. In the source computer room, the source service component only monitors the operation of the source client App_S writing the first business data to the source Redis cache Redis_S, without obtaining the first business data in the source Redis cache Redis_S. In the destination computer room, the destination synchronization component Sync_O can also directly write the first business data into the destination Redis cache Redis_O, without the need for the destination service component to first obtain the first business data and then write the first business data into the destination Redis cache Redis_O.

[0090] In some embodiments, different types of source servers Server_S and destination servers Server_O may use different synchronization methods. For example, when both the source server Server_S and the destination server Server_O are servers of the first type, the first method described above may be used to synchronize the first business data to the destination Redis cache Redis_O. When both the source server Server_S and the destination server Server_O are servers of the second type, the second method described above may be used to synchronize the first business data to the destination Redis cache Redis_O.

[0091] Among them, the first type of server can be a cloud server, and the second type of server can be a non-cloud server, but the present disclosure is not limited to this. The application scenario when the first type of server is a cloud server is a cloud scenario. In the cloud scenario, the source server Server_S and the destination server Server_O can both be Paas platforms, and the source synchronization component Sync_S and the destination synchronization component Sync_O can both be pulsar servers. PaaS is the abbreviation of (Platform as a Service), which refers to platform as a service. The business model of providing a server platform as a service, and the service provided by the program through the network is called SaaS (Software as a Service), which is one of the three service models of cloud computing, and the corresponding server platform or development environment in the cloud computing era is provided as a service to become PaaS. The source client App_S can monitor the operation of writing the first business data to the source Redis cache Redis_S, and send the first business data written to the source Redis cache Redis_S to the source service component in the source server Server_S through an asynchronous thread (http or grpc), and then the source service component writes the first business data to the source synchronization component Sync_S, thereby synchronizing to the destination synchronization component Sync_O. The disclosed embodiment utilizes the regional replication capability of the pulsar server to achieve data synchronization between different computer rooms in a cloud scenario.

[0092] The scenario when the second type of server is a non-cloud server is a non-cloud scenario. In the non-cloud scenario, the source server Server_S and the destination server Server_O can both be replicator servers, and the source synchronization component Sync_S and the destination synchronization component Sync_O can both be synchronization modules of the replicator server. The source service component in the source server Server_S can monitor the operation of writing the first business data to the source Redis cache Redis_S. If monitored, the source service component can notify the source synchronization component Sync_S to obtain the first business data written to the source Redis cache Redis_S, so that the source synchronization component Sync_S synchronizes the obtained first business data to the destination synchronization component Sync_O. The disclosed embodiment can disguise the replicator server as a slave node, and use the source server Server_S as the master node to synchronize data from the master node to the slave node, thereby realizing data synchronization between different computer rooms in the non-cloud scenario.

[0093] In a non-cloud scenario, the connection information of the target synchronization component Sync_O can be pre-configured to the source server Server_S, so that the source synchronization component Sync_S establishes a communication connection with the target synchronization component Sync_O based on the connection information of the target synchronization component Sync_O. The connection information may include an ID, a port number, etc., which are used to uniquely identify the target synchronization component Sync_O. The source server Server_S can generate an authentication identifier based on the connection information of the server and the connection information of the target server Server_O, and store the authentication identifier.

[0094] Figure 2 and Figure 3 Schematic diagrams showing the data synchronization process in cloud scenarios and non-cloud scenarios respectively. Figure 2In the cloud scenario, first, the source client App_S of IDC_S can receive the first business data and write the first business data into the source Redis cache Redis_S of IDC_S. The first business data written successfully is sent to the Paas platform of IDC_S. The source client of IDC_S can be developed in the Springboot-starter mode and integrated with the Redis-starter script. Its main functions are as follows: (1) Encapsulate the operation mode of each type of client among multiple types. When the client is initialized, the Redis-starter script can call the operation mode corresponding to the type based on the client type; (2) Call the lua script to write the received first business data into the Redis cache of IDC_S; (3) Use an asynchronous thread (for example, http or grpc) to write the received first business data into the PaaS platform of IDC_S (that is, the above-mentioned source server Server_S). The main functions of the source server Server_S are: further encapsulate the first business data sent by the client, and use the lua script to add or obtain relevant configuration information. Through the cooperation of the client and the server, the purpose of cache synchronization is achieved.

[0095] Then, after receiving the first business data, the PaaS platform of IDC_S can write the first business data to the pulsar server of IDC_S (i.e. the above-mentioned source synchronization component Sync_S), and the pulsar server of IDC_S synchronizes the first business data to the pulsar server of IDC_O (i.e. the above-mentioned destination synchronization component Sync_O) through regional replication. The PaaS platform can adopt the client-server development model to build PaaS services, developed using springboot-2.1.9.RELEASE, and deployed in a high-availability cluster. Furthermore, the PaaS platform of IDC_S can pre-acquire the identification information of the PaaS platform of IDC_O (i.e. the above-mentioned destination server Server_O), and generate an authentication identifier (accessKey) based on the identification information of the PaaS platform of IDC_O and the identification information of this PaaS platform. Before synchronizing data to the pulsar server of IDC_O, access authentication can be performed on the PaaS platform of IDC_O based on the authentication identifier. Only when the authentication is legal will the first business data be synchronized to the pulsar server of IDC_O through the pulsar server of IDC_S, thereby improving data security.

[0096] The disclosed embodiment also supports filtering of specified fields, that is, specifies which fields of the first business data do not need to be synchronized, and supports prefix matching mode. Specifically, the first business data carries the identification information of the specified field (for example, the field name), and the PaaS platform of IDC_S can pre-store the identification information of the first business data that does not need to be synchronized. After the client of the source computer room IDC_S sends the first business data to the PaaS platform of the source computer room IDC_S, the PaaS platform of the source computer room IDC_S can compare the identification information carried in each first business data with the pre-stored identification information. If the two are the same, the first business data is not synchronized; if the two are different, the first business data is sent to the pulsar server of the source computer room IDC_S, so as to synchronize the first business data to the Redis cache of the destination computer room IDC_O (that is, the above-mentioned destination Redis cache). In this way, the business data that needs to be synchronized can be specified, and unnecessary data synchronization can be reduced, thereby reducing the resource occupation during the synchronization process and improving the flexibility of the synchronization process.

[0097] After the PaaS platform of the destination computer room IDC_O monitors that the first business data is written to the pulsar server of the destination computer room IDC_O, the PaaS platform of the destination computer room IDC_O can synchronize the first business data to the Redis cache of the destination computer room IDC_O. In this way, cross-computer room data synchronization from the source computer room IDC_S to the destination computer room IDC_O is realized.

[0098] In some embodiments, each server may include a control terminal (control), a receiving terminal (produce), and a logical processing terminal (consume). The control terminal may generate a globally unique authentication identifier (accesskey) based on the access information of the source server Server_S and the destination server Server_O, perform access authentication based on the authentication identifier, generate a message queue (topic) on the pulsar server, receive the business data sent by the client, and provide multi-tenant functions, differentiate by tenants, and implement data isolation and other functions. When the server is used as the source server Server_S, the receiving terminal may write the business data received by the control terminal to the pulsar server. When the server is used as the destination server Server_O, the logical processing terminal may consume the business data synchronized from other computer rooms to the pulsar server, and write the business data to the Redis cache. In some embodiments, the server may adopt a mode in which the management terminal, the receiving terminal, and the logical processing terminal are separated to achieve the purpose of improving the performance of a single service, that is, the SaaSization of PaaS services.

[0099] The receiving end and the logic processing end can both set up their own caches (for example, memory caches), and the server can periodically retrieve the business data that has changed on the management end and synchronize it to the caches of the receiving end and the logic processing end. In this way, the dependence on the database is weak, and the changed links and other data sources can be operated to provide near-real-time change synchronization updates.

[0100] In some embodiments, the source server Server_S can also monitor and warn the synchronization results. Specifically, the source server Server_S can report monitoring data to a monitoring platform (e.g., Grafana), which can be used to report monitoring indicators such as synchronization failure, synchronization rate, and synchronization time.

[0101] After the business data is successfully sent to the PaaS platform, if an exception occurs, the abnormal result can be handled by using the CallBack method. In order to let the business party understand the error information in detail, a general exception handling method is defined here, which can be recorded in a class (called the exception handling class). Specifically, the exception handling class includes the method of operating Redis, the data requested by Redis, and the exception information.

[0102] In non-cloud scenarios, the server in one computer room can be disguised as a slave node and deployed in another computer room, allowing it to act as a slave of another node (such as Fig.16 As shown in the slave2 shown in the figure), when the data written by the two computer rooms arrives at the master node, the master node sends the PSYNC command to the slave node, and the slave node can receive the data from the two computer rooms. However, most Redis services in cloud scenarios use the VIP mode to provide external services. The VIP mode can be considered as a single-node mode of Redis by the business side. The VIP mode provides a unique interface host:port to the outside world. This host is a fixed IP. The server cannot disguise itself as a slave because it cannot use the above master-slave synchronization data logic, and it cannot monitor the data sent by the master to the slave node. Therefore, a node can only act as a slave node of a master, and slave nodes cannot be shared across computer rooms. Therefore, in the cloud scenario, the cross-regional data replication function of the pulsar server is used for data synchronization. For simplicity, the data synchronization between the two computer rooms IDC1 and IDC2 is used as an example in the figure. It can be understood that in actual applications, data synchronization can also be performed between more than two computer rooms.

[0103] See also Figure 3In a non-cloud scenario, first, the source client App_S of IDC_S can write the first business data into the source Redis cache Redis_S of IDC_S. The replicator service component of IDC_S (represented as the source service component in the figure) can monitor the operation of writing the first business data to the Redis cache of IDC_S in real time. If monitored, the first business data is synchronized to the async module of IDC_O (i.e. the above-mentioned destination synchronization component Sync_O) through the async module of IDC_S (i.e. the above-mentioned source synchronization component Sync_S). After monitoring that the async module of IDC_O is written with the first business data, the replicator service end of IDC_O (represented as the destination service component in the figure) can notify the async module of IDC_O to write the first business data into the Redis cache of IDC_O (i.e. the above-mentioned destination Redis cache).

[0104] In non-cloud scenarios, two different synchronization modes, full synchronization and non-full synchronization, are also supported. Specifically, a configuration item can be added to indicate the synchronization mode. In some embodiments, when the value of the configuration item is the first value, only incremental synchronization is supported. That is, after the server is started, it only receives the latest written data, regardless of the previously written data. If the value of the configuration item is the second value, full synchronization can be supported.

[0105] In some embodiments, the destination server Server_O can send the generated Replication ID and offset to the source server Server_S, and the source server Server_S determines whether the Replication ID is legal. If so, the offset is used to determine whether there is any synchronized data for this ID. When the synchronization mode is full synchronization, the offset is 0. When incremental synchronization is selected, the destination server Server_O will query the offset of this Replication ID from the source Redis cache Redis_S. After the connection, the offset is directly reported to the source server Server_S. The source server Server_S can determine whether the reported offset is consistent with the offset of the corresponding Replication ID stored in itself. If they are consistent, they will not be synchronized, otherwise they will be synchronized.

[0106] In some embodiments, the type of the source server Server_S can be configured through a configuration file so that the source client App_S can determine the type of the source server Server_S. The configuration file may include a call address. If the call address is empty, the type of the source server Server_S is the first type. If the call address is not empty, the type of the source server Server_S is the second type.

[0107] Specifically, the source client App_S can introduce Redis-starter (a Java program building tool) through Maven or Gradle, configure the necessary parameters in the configuration file, and during the startup of the source client App_S, according to the characteristics of springboot loading starter, first, springboot searches for the configuration file under the classpath under the project directory, then loads the configuration class according to the configuration file configuration, and then automatically configures and injects the Bean into the Spring context according to the preset annotation conditions in the configuration class. During the loading process of the Bean, the configuration file information will be obtained. After obtaining the configuration file information, the source client App_S will be initialized according to the configured data. Figure 7 As shown in the figure, when the redis client connection is initialized, the address, port, password, etc. of Redis will be checked to complete the initial connection work of the client selection. The figure shows that the client includes three different types of Jedis, Redission and Lettuce, but it is not limited to this in actual applications. The source client App_S can call the method that matches the type of this client for initialization. After initialization, the information of the configuration file has been assigned to the specific class object. According to the characteristics of spring, this object is a singleton, so it does not need to be initialized every time the method is called later, resulting in the opening of Java heap memory and GC recovery. The type of the source server Server_S can be determined according to the configured call address.

[0108] After determining the type of the source server Server_S, you can call the wrapper class required by the source server Server_S, assemble the business data, and then write it to the source Redis cache Redis_S. For example, see Figure 8 After determining that the source server Server_S is a PaaS platform, the synchronization data sending method can be identified based on whether the prefix of the configuration address information is http: / / or grpc: / / to complete the data synchronization operation.

[0109] In some embodiments, see Figure 4, there is a data loop problem during the synchronization process. After the first business data is synchronized from IDC_S to IDC_O, the server of IDC_O monitors that new business data is written to IDC_O, and executes the logic in the aforementioned embodiment to write the business data (that is, the first business data synchronized from IDC_S), thereby resynchronizing the business data to IDC_S. In this way, a piece of business data will be constantly synchronized between the two computer rooms, causing a data loop problem. The figure takes a non-cloud scenario as an example. It can be understood that in the cloud scenario, there is also a data loop problem. In order to solve this problem, the source of the data can be judged. If it is data synchronized from other computer rooms, it is written to the redis service of this computer room; if it is not data synchronized from other computer rooms, it is discarded.

[0110] Specifically, the first business data may carry the computer room identifier of the computer room that sends the first business data (i.e., the source computer room IDC_S). After the source server Server_S monitors the operation of writing the first business data into the source synchronization component Sync_S, the computer room identifier carried in the first business data (i.e., the computer room identifier of the source computer room IDC_S) can be parsed, and the computer room identifier carried in the first business data can be compared with the computer room identifier of the source computer room IDC_S. Since the parsed computer room identifier is the same as the computer room identifier of the source computer room IDC_S, it can be determined that the source of the data is the source computer room IDC_S, so that the operation of writing the first business data into the source Redis cache Redis_S is not performed. Similarly, after the destination server Server_O monitors the operation of writing the first business data into the destination synchronization component Sync_O, it can parse the computer room identifier carried in the first business data (i.e., the computer room identifier of the source computer room IDC_S) and compare it with the computer room identifier of the destination computer room IDC_O. Since the parsed computer room ID is different from the computer room ID of the destination computer room IDC_O, the destination server Server_O will execute the operation of writing the first business data into the destination Redis cache. A field "aaa" can be defined to record the computer room ID carried in the first business data. By parsing this field, it can be determined which computer room sent the first business data.

[0111] In some embodiments, in a non-cloud scenario, the number of source servers Server_S may be greater than or equal to 1. When the number of source servers Server_S is greater than 1, after data is written to the source Redis cache Redis_S, each source server Server_S can monitor the write operation to the source Redis cache Redis_S in real time, and synchronize the first business data in the source Redis cache Redis_S to the destination synchronization component Sync_O through the source synchronization component Sync_S. In this way, the same business data will be synchronized multiple times, thereby increasing network overhead and affecting IO performance. The above phenomenon is called repeated consumption of Redis cache data.

[0112] In order to solve this problem, a service component identification information can be pre-assigned to each source server Server_S to identify one of the multiple first servers. Different source servers Server_S are assigned different service component identification information. In addition, the first business data of the source Redis cache Redis_S written to the source computer room IDC_S also carries the service component identification information. Each source server Server_S can compare the service component identification information assigned to the server with the service component identification information carried in the first business data. If the two are consistent, the first business data in the source Redis cache Redis_S is synchronized to the destination synchronization component Sync_O through the source synchronization component Sync_S. If they are inconsistent, the operation is not performed.

[0113] See also Figure 5 , you can define the field "bbb" and define an array to store a set of values, for example, {1,2,3,…,30}. When writing the first business data to the source Redis cache Redis_S, randomly pick a number from the array and assign it to "bbb". When the source server Server_S starts, it can first register itself with the etcd registration center to obtain the service component identification information of the source server Server_S (remembered as key), which is specifically named as follows: / service name / computer room identification / uuid. For example, / sonic-replicator / idcname / 1. The watch function of the etcd registration center can monitor the startup and offline source server Server_S in real time, and define a globally unique static array to store the generated instance key.

[0114] Secondly, one or more values ​​can be obtained from the array for each source server Server_S based on a preset rule, and added to the list NODE_LIST of the source server Server_S, wherein NODE_LIST is a globally unique static array, and the lists NODE_LIST of different source servers Server_S include different values. The preset rule may be to evenly distribute the values ​​in the array. For example, Figure 6 As shown, assuming that the array includes 30 numbers from 1 to 30, and the number of source server Server_S is 3 (each source server Server_S is denoted as R1, R2 and R3), then the NODE_LIST of each source server Server_S may include 10 numbers, for example, the NODE_LIST of the source server Server_SR1 includes the values ​​1 to 10 in the array, the NODE_LIST of the source server Server_SR2 includes the values ​​11 to 20 in the array, and the NODE_LIST of the source server Server_SR3 includes the values ​​21 to 30 in the array. It can be understood that this is only an exemplary description. In actual applications, the number of values ​​in the array may be greater than or less than 30, and the values ​​in the array may be discontinuous, the number of source server Server_S may be other numbers, and the method of generating NODE_LIST is not limited to evenly distributing the values ​​in the array.

[0115] After the source server Server_S receives the first business data, it can parse the value of the encapsulation class "bbb" (that is, the second marking parameter mentioned above), and then query the value included in the NODE_LIST of this server (that is, the first marking parameter mentioned above). If the two are consistent, the source server Server_S executes the operation of synchronizing the first business data in the source Redis cache Redis_S to the destination synchronization component Sync_O through the source synchronization component Sync_S.

[0116] In some embodiments, if the service component identification information corresponding to at least one source server Server_S changes, each source server Server_S will be reallocated the service component identification information. Before the service component identification information is reallocated, the first business data written to the source Redis cache Redis_S of the source computer room IDC_S can be written into the blocking queue first. After the service component identification information is reallocated, the source server Server_S can compare the service component identification information reallocated by the server with the service component identification information carried in the first business data. If the two are consistent, the source synchronization component Sync_S in the source server Server_S obtains the first business data from the blocking queue and synchronizes it to the destination synchronization component Sync_O. When the source synchronization component Sync_S obtains the first business data, a separate thread can be restarted to consume the first business data in the blocking queue. The allocation rules for reallocating the service component identification information can be the same as the preset rules of the aforementioned embodiment, for example, they are all evenly distributed. The reallocated service component identification information can be stored in etcd, which is an open source, highly available distributed key-value storage system that can be used to configure the registration and discovery of sharing and services.

[0117] In a distributed environment, the order of two events is generally determined by system time. However, because the system time of each service is verified through the ntp protocol, there is an error of 1ms when ntp synchronizes the time of each server in the local area network, and in the wide area network, there is a difference of tens of milliseconds. Therefore, using system time to determine the order of time in two different computer rooms is inaccurate. In some embodiments, after the business data generated by the client of IDC_S is written into the Redis cache of IDC_S, the client of IDC_O also generates business data in near real time. When the two computer rooms write business data successively, if there is no priority judgment, it may cause the data in the Redis cache of the two computer rooms to be inconsistent, thereby causing problems such as business processing errors. See Fig. 9, the business data written in the IDC_S computer room has key=name, value=v1, and the data is synchronized from the IDC_S computer room to the IDC_O computer room through the synchronization link. Almost at the same time, the business data written in the IDC_O computer room has key=name, value=v2, and the data is synchronized from the IDC_O computer room to the IDC_S computer room through the synchronization link. Finally, the data in the Redis cache of the IDC_S computer room changes to key=name, value=v2 in response to the synchronization operation from the IDC_O computer room to the IDC_S computer room, while the data in the Redis cache of the IDC_O computer room changes to key=name, value=v1 in response to the synchronization operation from the IDC_S computer room to the IDC_O computer room. This results in the business data in the Redis cache of the two computer rooms being out of sync.

[0118] In order to solve this problem, the source vector clock and source system clock corresponding to the first business data can be synchronized to the destination synchronization component Sync_O through the source synchronization component Sync_S. The source vector clock and source system clock synchronized to the destination synchronization component Sync_O are used by the destination server Server_O to determine whether to write the first business data synchronized to the destination synchronization component Sync_O into the destination Redis cache.

[0119] The logical clock is used to determine the order in which different events are executed. The logical clock of the source computer room IDC_S can be recorded in the source Redis cache Redis_S, and the logical clock of the destination computer room IDC_O can be recorded in the destination Redis cache. The initial value of each logical clock (source logical clock or destination logical clock) is 0. After a write operation is performed to the Redis cache of a computer room, the value of the logical clock in the Redis cache of the computer room is increased by 1. The larger the value of the logical clock, the later the corresponding event occurs. Therefore, according to the Last Write Wins (LWW) principle, if the value of the source vector clock is greater than the value of the destination vector clock in the destination Redis cache, it means that the last write operation of the source computer room IDC_S to the Redis cache is after the last write operation of the destination computer room IDC_O to the destination Redis cache, so that the first business data of the source computer room IDC_S can be written to the destination Redis cache. On the contrary, if the value of the source vector clock is smaller than the value of the destination vector clock in the destination Redis cache, it means that the last write operation of the source computer room IDC_S to the Redis cache was before the last write operation of the destination computer room IDC_O to the destination Redis cache, thereby refusing to write the first business data of the source computer room IDC_S to the destination Redis cache.

[0120] See also Fig.10, you can define the field "ccc", each field has its own "ccc" value. When the service is used for the first time, the value of this field is ccc=0. During each synchronization, first check the value of the "ccc" field in the Redis cache of the current computer room, then add 1, assemble the data, and rewrite it into the Redis cache of the current computer room. Fig.10 As shown, when event X and event Y collide at Y, because the value of "ccc" of the computer room where event X is located is 2, and the value of "ccc" of the computer room where event Y occurs is 5, when the data generated by event X is synchronized to the computer room where event Y occurs, the value of "ccc" of event Y is greater than the value of "ccc" of event X, then the synchronization is abandoned because event Y occurs last, according to the LWW principle.

[0121] In some embodiments, if the value of the source vector clock is equal to the value of the destination vector clock, whether to write the first business data to the destination Redis cache can be further determined based on the value of the source system clock and the value of the destination system clock in the destination Redis cache. If the value of the source system clock is greater than the value of the destination system clock in the destination Redis cache, then write, otherwise refuse to write.

[0122] Furthermore, the priorities of different computer rooms can be set. If the value of the source vector clock is equal to the value of the destination vector clock, and the value of the source system clock is equal to the value of the destination system clock, it can be further determined whether to write the first business data into the destination Redis cache based on the priority of the source computer room IDC_S and the priority of the destination computer room IDC_O. If the priority of the source computer room IDC_S is higher than the priority of the destination computer room IDC_O, then write, otherwise refuse to write.

[0123] In some embodiments, after reading the value of the source vector clock, the value of the source vector clock can be cached in a memory cache (memoryCache). When the business aspect operates the key in the source Redis cache Redis_S, the key value is updated by adding 1 to the value of the key, and then the current timestamp of the system (i.e., the source system clock) is obtained, and the field name of the source vector clock is used as the key. The value of the source vector clock and the value of the source system clock form a hash value, and the hash value is used as the value and written into the source Redis cache Redis_S together. In this way, the problem of inconsistent timestamps of the two computer rooms when directly using the system timestamp for comparison is reduced. At the same time, the hash function of redis is used here. On the one hand, the amount of data that can be stored is large; on the other hand, since Redis adopts a key-value architecture and can only store two fields, by performing hash processing, the field name of the source vector clock, the value of the source vector clock, and the value of the source system clock can be stored at the same time. Further, the expiration time of this key can also be set (for example, 30 minutes). Expired keys and corresponding values ​​can be cleared from the source Redis cache Redis_S, thereby reducing the space occupied by the source Redis cache Redis_S.

[0124] In some embodiments, all data starting with sonic:vck in the Redis cache can be queried to obtain the value of the vector clock, and Caffeine (an efficient cache framework) is used to cache the value of the vector clock, and the expiration time is 30 minutes. This can reduce multiple interactions with the Redis cache during data synchronization and solve the problem of adding 1 to the existing value in the Redis cache. The specific processing logic is as follows Fig.17 After the client is initialized, the data starting with sonic:vck in the Redis cache can be scanned. If the data is found, the scanned value is determined as the current value of the vector clock and the scanned value is cached in Caffeine. If the data is not found, the current value of the vector clock is determined as 0 and 0 is cached in Caffeine.

[0125] The fields such as "aaa", "bbb", "ccc" used in the above embodiment can be encapsulated into a starter, that is, the Redis-starter in the above embodiment. This starter encapsulates and stores the input parameters, and the input parameters can be recorded in the parameter class. The input parameters in the parameter class may include at least the following parts: parameters for solving the loop problem, parameters for solving the event sequence problem, parameters for solving the repeated consumption problem, and parameters for setting the expiration time.

[0126] According to the above parameter class, the first business data written will eventually be stored in redis hash mode. Redis uses key-value mode for data storage and can only store two fields. Through the above method, multiple fields can be encapsulated and stored together, thus solving the problem that Redis cannot store multiple fields. In addition, the field values ​​of each field and the first business data can be written to the source Redis cache Redis_S through the Lua script to ensure atomicity.

[0127] When the first server and the second server are of the first type or the second type, different parameter classes can be generated. For example, when the first server and the second server are both of the first type (for example, in a cloud scenario), the parameter class may include two fields of "aaa" and "ccc"; when the first server and the second server are both of the second type (for example, in a non-cloud scenario), the parameter class may include three fields of "aaa", "bbb" and "ccc". In addition, other fields can be added to the parameter class according to actual needs, which will not be repeated here. The configuration information of the client can be configured so that the client determines the type of the server, thereby calling the parameter class that matches the type of the server, and generating the first business data based on the called wrapper class. In this way, the synchronization method of the embodiment of the present disclosure can automatically adapt to different application scenarios.

[0128] When solving the problem of conflict between two or more pieces of data, relational databases can add a new version number field or a unique sequence, but Redis is a key-value database and cannot add extra fields. When adding an auxiliary key, various issues such as atomicity, consistency, and persistence must be considered. This brings many problems to cache synchronization.

[0129] Although during cache synchronization, the destination server Server_O can determine whether the operation of the current computer room is executed correctly based on the execution result or exception when writing the data synchronized from the source server Server_S, it cannot perceive the data of other computer rooms, such as Fig.11As shown in the figure, when IDC1 consumes business data key=k, value=1 from the message queue, it is written into the Redis cache of IDC1. At this time, there is no data in the Redis cache of the IDC1 computer room. The vector clock of the synchronized business data is 1, which is greater than the vector clock of the computer room, which is 0, so it is successfully written. The same is true for IDC2. When the synchronized data is sent to IDC3, during this synchronization process, the server in IDC3 writes a piece of business data with key=k, value=2, and the event of writing the business data with key=k, value=2 occurs later than the event of writing the business data with key=k, value=1. At this time, the value of the vector clock is 1, and the value of the vector clock carried by the synchronized data key=k, value=1 is also 1. The two values ​​are equal, so it is necessary to compare the size of the system clock again. Because the event with value=2 occurs later, and the event with value=1 occurs earlier, the priority of the event with value=2 is greater than the priority of the event with value=1, and the synchronization fails. As a result, the data in the Redis cache of the three IDCs is inconsistent, and the monitoring and alarm of the IDC3 computer room only reports that a data write failure occurred in the current computer room, and cannot perceive the specific values ​​of other computer rooms.

[0130] Based on this, it is necessary to verify whether the data in each computer room is consistent after data synchronization. When performing verification, the relevant technology generally gives a source Redis connection information and a target Redis connection information, starts the function, queries the source Redis's comparison key into the memory, and then queries the target Redis's key, and then compares the two key types, value sizes, lengths, and types to see if they are consistent. The above method is not friendly to incremental comparison support. The Redis cache synchronization consistency comparison solution compares the data of multiple computer rooms in near real time. Therefore, it is necessary to know which data keys have been synchronized within a certain period of time, collect these keys, and compare them in batches. However, when performing batch comparison, the relevant technology has problems such as low efficiency and low accuracy. In addition, the relevant technology uses Redis's own scan command internally during comparison, and needs to query the source Redis and the target Redis, which may occupy the Redis thread, resulting in delays or errors when the business being executed queries Redis.

[0131] Based on this, the embodiment of the present disclosure provides another consistency verification method. Fig.12, the source computer room IDC_S also includes a first verification component, and the destination computer room IDC_O also includes a second verification component. For the sake of distinction, the first verification component is referred to as the source verification component below, and the second verification component is referred to as the destination verification component. The source verification component is communicatively connected to the destination verification component. The destination verification component can obtain the first business data synchronized to the destination synchronization component Sync_O, and synchronize the obtained first business data to the source verification component. The source verification component can obtain the first business data written to the source Redis cache through the source synchronization component Sync_S, and perform consistency verification on the first business data synchronized to the destination synchronization component based on the first business data obtained from the destination verification component and the first business data obtained from the source synchronization component Sync_S.

[0132] Among them, the source verification component and the destination verification component can communicate through http or grpc and other methods. The connection information of the source verification component can be pre-configured to the destination verification component so that the source verification component and the destination verification component establish a communication connection. The communication link between any source computer room IDC_S and any destination computer room IDC_O is called a synchronization link, and each communication link can correspond to a message queue. The destination verification component can subscribe to the first business data from the message queue corresponding to the synchronization link in the destination synchronization component Sync_O based on the synchronization link between the source computer room IDC_S and the source computer room IDC_S, and send the subscribed first business data to the source verification component of the source computer room IDC_S.

[0133] When the number of source computer room IDC_S is m and the number of destination computer room IDC_O is n, the number of synchronization links is m*n. For example, assuming that the destination computer room IDC_O includes computer room A, computer room B and computer room C, and the source computer room IDC_S includes computer room D and computer room E, the synchronization links include the synchronization link from computer room D to computer room A (referred to as link DA), the synchronization link from computer room D to computer room B (referred to as link DB), the synchronization link from computer room D to computer room C (referred to as link DC), the synchronization link from computer room E to computer room A (referred to as link EA), the synchronization link from computer room E to computer room B (referred to as link EB), and the synchronization link from computer room E to computer room C (referred to as link EC). Each of the above synchronization links corresponds to a different message queue.

[0134] Taking computer room A as an example, the synchronization links of computer room A are link DA and link EA. Therefore, the destination verification component of computer room A can subscribe to the first business data synchronized from computer room D to computer room A from the message queue corresponding to link DA in the destination synchronization component Sync_O (referred to as queue 1), and subscribe to the first business data synchronized from computer room E to computer room A from the message queue corresponding to link EA in the destination synchronization component Sync_O (referred to as queue 2). Then, the destination verification component of computer room A can send the first business data subscribed from queue 1 to the source verification component of computer room D, and send the first business data subscribed from queue 2 to the source verification component of computer room E. The operations performed by computer rooms B and C are similar and will not be repeated here.

[0135] Similarly, the source verification component of computer room D can also subscribe to the first business data synchronized from computer room D to computer room A in the message queue corresponding to link DA in the source synchronization component Sync_S of computer room A, and compare the subscribed first business data with the first business data sent by the destination verification component of computer room A, so as to determine whether the first business data synchronized from computer room D to computer room A is consistent with the first business data of this computer room.

[0136] In some embodiments, during the data synchronization process, the source server Server_S can send the first business data to different message queues in the source synchronization component Sync_S according to the synchronization link. After the source synchronization component Sync_S receives the first business data, the source verification component can obtain the first business data from the corresponding message queue of the source synchronization component Sync_S and cache it. Similarly, during the data synchronization process, the source synchronization component Sync_S can send the first business data to the queue message queue of the destination synchronization component Sync_O according to the synchronization link. After the destination synchronization component Sync_O receives the first business data, the destination verification component can obtain the first business data from the corresponding message queue of the destination synchronization component Sync_O.

[0137] Based on the multi-subscription feature of the message queue, the data in the message queue can be consumed in batches and then compared with the data in other computer rooms. For example, 1,000 pieces of data in the message queue can be consumed each time. The solution of the disclosed embodiment does not need to repeatedly obtain the connection information of the source Redis and the destination Redis, which improves the efficiency of consistency verification.

[0138] Optionally, the first business data obtained by the source verification component from the destination verification component and the first business data obtained by the source verification component from the source synchronization component Sync_S are both stored in the etcd of the source computer room IDC_S, and the source verification component reads the data from the etcd of the source computer room IDC_S and performs consistency verification. By caching the data to be verified in etcd, there is no need to interact with the Redis cache during the verification process, thereby avoiding frequent operations on the Redis cache during the verification process and reducing the impact on normal business.

[0139] In some embodiments, the consistency check is used to check the key of the first business data obtained from the destination check component and the key of the first business data obtained from the source synchronization component Sync_S. The value of the first business data stored in etcd of the source computer room IDC_S is set to a preset value, and the storage space occupied by the preset value is less than the storage space occupied by the actual value of the first business data.

[0140] In this embodiment, only the key of the first business data can be verified without verifying the value of the first business data. Therefore, the value of the first business data can be replaced with a preset value to make the space occupied by the first business data stored in etcd smaller, thereby saving etcd space. Among them, the preset value can be a 1-bit binary number such as 0 or 1, or other values. Furthermore, an expiration time (for example, 5 minutes) can also be set for the key in etcd. Expired keys can be automatically deleted from etcd, thereby reducing the space occupied by etcd.

[0141] In other embodiments, the consistency check is also used to verify the value of the first business data obtained from the destination verification component and the value of the first business data obtained from the source synchronization component Sync_S based on the source vector clock and the first system clock of the source computer room IDC_S and the second vector clock and the second system clock of the destination computer room IDC_O.

[0142] In this embodiment, both the key of the first business data and the value of the first business data can be verified. Since Redis supports multiple types of data such as strings, linked lists, sets, ordered sets, hashes, etc., when verifying the value of the business data, it is necessary to consider the type of business data, which makes the verification process more complicated. In order to simplify the value verification process, the embodiment of the present disclosure performs verification directly based on the vector clock and the system clock. If the source vector clock is the same as the destination vector clock, and the source system clock is the same as the destination system clock, it is determined that the value of the first business data sent by the destination verification component is consistent with the value of the first business data obtained from the source synchronization component Sync_S. If the source vector clock is different from the destination vector clock, or the source system clock is different from the destination system clock, it is determined that the value of the first business data sent by the destination verification component is inconsistent with the value of the first business data obtained from the source synchronization component Sync_S.

[0143] Through the above method, it is possible to verify the key alone and the key and value separately. The verification method can be configured through configuration information, so that different verification methods can be selected in different situations, which improves the flexibility of the verification method.

[0144] See also Fig.13 If the source verification component determines that there is a difference (i.e., inconsistency) between the first business data sent by the destination verification component and the first business data obtained from the source synchronization component Sync_S, the source verification component can record the difference data and report the difference data to the monitoring platform so that the monitoring platform can issue a monitoring alarm. Alternatively, the source verification component can also return the verification result to the destination verification component so that the destination verification component reports the verification result to the monitoring platform for monitoring alarm. If the source verification component determines that there is no difference (i.e., consistency) between the first business data sent by the destination verification component and the first business data obtained from the source synchronization component Sync_S, the comparison ends.

[0145] In the cloud scenario, when synchronizing data in multiple computer rooms, there may be n (n is a positive integer) servers connected. These n servers achieve high availability with the help of gateway load balancing. The overall flow chart is as follows: Fig.14As shown, the gateway sends different synchronization requests (for example, synchronization request 1, synchronization request 2, and synchronization request 3) to different PaaS platforms for processing. This means that there are n synchronization links. When all servers are synchronizing data, it will cause competition for CPU, IO and other resources in a large case, resulting in some services running efficiently and some services running inefficiently. In order to reasonably and fairly allow each service to run normally without interfering with each other, and considering the operating efficiency, maintenance and other aspects, the embodiment of the present disclosure proposes a solution for Paas service SaaS. After the service is SaaS-ized, the service can be deployed on its own server or virtual machine, and there is no need to compete for resources, which solves the problem of insufficient service resources and the high synchronization time.

[0146] Specifically, when the source server Server_S and the destination server Server_O are both first type servers (for example, in a cloud scenario), the number of synchronization links can be greater than or equal to 1. If the number of synchronization links is greater than 1, different synchronization links include different source servers Server_S or different destination servers Server_O, and at least two synchronization links correspond to different tenants, then the source synchronization component Sync_S may include multiple message queues, and different message queues correspond to different synchronization links; wherein each synchronization link corresponds to a tenant identifier, which is used to determine the tenant corresponding to the destination computer room IDC_O on the synchronization link. Then, the target tenant to which the first business data belongs can be determined, the target synchronization link corresponding to the target tenant can be determined, and the received first business data can be written into the target message queue corresponding to the synchronization link corresponding to the target tenant, so as to write the first business data into the destination synchronization component Sync_O of the destination computer room IDC_O on the synchronization link corresponding to the target tenant through the target message queue.

[0147] For example, assuming that there are synchronous links 1 to 7, wherein synchronous link 1 and synchronous link 2 correspond to tenant 1, and synchronous link 3 to synchronous link 7 correspond to tenant 2. Assuming that the first business data received is the business data of tenant 1, it can be written into the target message queue corresponding to synchronous link 1 and the target message queue corresponding to synchronous link 2, thereby writing the first business data into the destination synchronization component Sync_O of the destination computer room IDC_O on synchronous link 1 through the target message queue corresponding to synchronous link 1, and writing the first business data into the destination synchronization component Sync_O of the destination computer room IDC_O on synchronous link 2 through the target message queue corresponding to synchronous link 2.

[0148] See also Fig.15, the PaaS platform management interface adds a custom identification field to the synchronization link, and this field is unique under the current tenant. After the configuration is completed, the value of the custom identification field is configured to the jvm parameter or the system file according to the agreement, so that when the service starts, this value can be obtained from two sources (configuration file or jvm parameter), and then the topic message is queried and stored in the database (DB), and then the topic information is initialized to the message queue of the pulsar server to generate a consumer object, which monitors the synchronized data in real time and synchronizes it to the corresponding Redis cache. At this point, the service SaaS is completed, and one management end is retained, and the synchronization data link is used normally. Furthermore, the priority can be set for the configuration file and jvm parameter, and the above custom identification field can be determined based on the priority. In the above way, the SaaS of PaaS service is realized. After the SaaS service is deployed to the computer room of this business, this service runs independently and is no longer confused with other synchronization links, which improves the service operation efficiency.

[0149] See also Fig.18 The embodiment of the present disclosure also provides a data synchronization method, which is applied to a second server in a second computer room, wherein the second server includes a second synchronization component, and the second synchronization component is communicatively connected with a first synchronization component in a first computer room; the method includes:

[0150] Step S21: monitoring the operation of synchronizing the first business data to the second synchronization component; after the first business data is written into the first cache of the first computer room by the first client of the first computer room, it is acquired by the first server of the first computer room and synchronized to the second synchronization component through the first synchronization component;

[0151] Step S22: if detected, the first business data synchronized to the second synchronization component is written into the second cache of the second computer room.

[0152] Similar to the method embodiment of the first server applied to the first computer room, in this embodiment, the first computer room may also be the source computer room IDC_S, and the second computer room may also be the destination computer room IDC_O. Thus, the first server, the first synchronization component, the first client, and the first cache of the first computer room may be respectively referred to as the source server Server_S, the source synchronization component Sync_S, the source client App_S, and the source cache Cache_S, and the second server, the second synchronization component, the second client, and the second cache of the second computer room may be respectively referred to as the destination server Server_O, the destination synchronization component Sync_O, the destination client App_O, and the destination cache Cache_O.

[0153] In some embodiments, the first business data carries the computer room identifier of the computer room that sends the first business data; and the first business data synchronized to the second synchronization component is written into the second cache of the second computer room, including: if the computer room identifier carried in the first business data is the same as the computer room identifier of the second computer room, the first business data is not written into the second cache; and / or if the computer room identifier carried in the first business data is different from the computer room identifier of the second computer room, the first business data is written into the second cache.

[0154] In some embodiments, the method also includes: monitoring a first vector clock and a first system clock synchronized to the second synchronization component, the first vector clock and the first system clock being the vector clock and the system clock corresponding to the first business data; obtaining the second vector clock and the second system clock in the second cache; and judging whether to write the first business data synchronized to the second synchronization component into the second cache based on the first vector clock, the first system clock, the second vector clock and the second system clock.

[0155] In some embodiments, the second vector clock and the second system clock are stored in the second cache in the form of a key-value pair, wherein the field name of the second vector clock serves as the key of the key-value pair, the value of the second vector clock and the value of the second system clock are hashed as the value of the key-value pair, and the value of the second vector clock is updated according to a preset update method.

[0156] The specific details of the method of the embodiment of the present disclosure are detailed in the aforementioned method embodiment applied to the first server in the first computer room, which will not be repeated here.

[0157] See also Fig.19 The embodiment of the present disclosure also provides a data synchronization device, which is applied to a first server in a first computer room, wherein the first server includes a first synchronization component, and the first synchronization component is communicatively connected with a second synchronization component of a second server in a second computer room; the device includes:

[0158] An acquisition module 110 is used to acquire first business data written by a first client in the first computer room into a first cache in the first computer room;

[0159] A synchronization module 120, configured to synchronize the first service data to the second synchronization component through the first synchronization component;

[0160] The first business data synchronized to the second synchronization component can be written into the second cache of the second computer room through the second server, and data is isolated between the first cache and the second cache.

[0161] The specific details of the embodiment of this device can be found in the aforementioned method embodiment applied to the first server in the first computer room, which will not be repeated here.

[0162] See also Fig. 20 The embodiment of the present disclosure also provides a data synchronization device,

[0163] A second server end applied to a second computer room, wherein the second server end includes a second synchronization component, and the second synchronization component is communicatively connected with the first synchronization component of the first computer room; the device includes:

[0164] The monitoring module 210 is used to monitor the operation of synchronizing the first business data to the second synchronization component; after the first business data is written into the first cache of the first computer room by the first client of the first computer room, it is acquired by the first server of the first computer room and synchronized to the second synchronization component through the first synchronization component;

[0165] The writing module 220 is used to write the first business data synchronized to the second synchronization component into the second cache of the second computer room if monitored.

[0166] The specific details of the embodiment of this device can be found in the aforementioned method embodiment applied to the second server in the second computer room, which will not be repeated here.

[0167] An embodiment of the present disclosure further provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0168] Fig.21 A more specific hardware structure diagram of a computing device provided by an embodiment of the present disclosure is shown, and the device may include: a processor 2101, a memory 2102, an input / output interface 2103, a communication interface 2104, and a bus 2105. The processor 2101, the memory 2102, the input / output interface 2103, and the communication interface 2104 are connected to each other through the bus 2105 in the device.

[0169] The processor 2101 can be implemented by a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided by the embodiments of the present disclosure. The processor 2101 can also include a graphics card, which can be a Nvidia titan X graphics card or a 10120Ti graphics card, etc.

[0170] The memory 2102 may be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. The memory 2102 may store an operating system and other application programs. When the technical solution provided by the embodiment of the present disclosure is implemented by software or firmware, the relevant program code is stored in the memory 2102 and is called and executed by the processor 2101.

[0171] The input / output interface 2103 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0172] The communication interface 2104 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0173] The bus 2105 includes a path that transmits information between the various components of the device (eg, the processor 2101 , the memory 2102 , the input / output interface 2103 , and the communication interface 2104 ).

[0174] It should be noted that, although the above device only shows the processor 2101, the memory 2102, the input / output interface 2103, the communication interface 2104 and the bus 2105, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0175] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0176] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0177] See also Fig. 22 The disclosed embodiment also provides a distributed data processing system, including: a first server, a first client and a first cache arranged in a first computer room, and a second server and a second cache arranged in a second computer room; the first server includes a first synchronization component, the second server includes a second synchronization component, and the first synchronization component is communicatively connected with the second synchronization component; the first client is used to write first business data to the first cache; the first server is used to obtain the first business data written to the first cache, and synchronize the first business data to the second synchronization component through the first synchronization component; the second server is used to write the first business data synchronized to the second synchronization component into the second cache, and the data between the second cache and the first cache is isolated.

[0178] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure or some parts of the embodiments.

[0179] The systems, devices, modules or units described in the above embodiments may be implemented by computer devices or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0180] Each embodiment in the present disclosure is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0181] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present disclosure.

Claims

1. A data synchronization method, characterized in that: A first server end is applied to a first computer room, the first server end includes a first synchronization component, and the first synchronization component is communicatively connected with a second synchronization component of a second server end in a second computer room; the method includes: Acquire first business data written by a first client in the first computer room into a first cache in the first computer room; Synchronizing the first business data to the second synchronization component through the first synchronization component; The first business data synchronized to the second synchronization component can be written into the second cache of the second computer room through the second server, and data is isolated between the first cache and the second cache.

2. The method according to claim 1, characterized in that: The first server and the second server are both first type servers, the first server further includes a first service component, and the second server further includes a second service component; the step of obtaining the first business data written by the first client of the first computer room into the first cache of the first computer room includes: Acquire, through the first service component, first business data written by the first client to the first cache; Writing the first business data acquired by the first service component into the first synchronization component; The first business data synchronized to the second synchronization component can be acquired by the second service component and written into the second cache by the second service component.

3. The method according to claim 2, characterized in that Writing the first business data acquired by the first service component into the first synchronization component includes: Determining a target tenant to which the first business data belongs; Determine a target synchronization link corresponding to the target tenant, where different synchronization links are data transmission links between different first computer rooms and / or different second computer rooms; The first business data is written into a target message queue corresponding to the target synchronization link in the first synchronization component; wherein the first synchronization component includes at least one message queue, and each message queue in the at least one message queue corresponds to a synchronization link.

4. The method according to claim 2, characterized in that: The first client includes a configuration file, and the configuration file includes a calling address; if the calling address is empty, the first server is a server of the first type.

5. The method according to claim 1, characterized in that The first server and the second server are both second-type servers, and the first server further includes a first service component; and obtaining the first business data written by the first client of the first computer room into the first cache of the first computer room includes: Monitoring, by the first service component, an operation of the first client writing first business data into the first cache; If detected, obtaining the first service data from the first cache through the first synchronization component; The first business data synchronized to the second synchronization component can be written into the second cache by the second synchronization component.

6. The method according to claim 5, characterized in that The number of the first service terminals is greater than 1, and the first service data carries service component identification information for identifying one of the multiple first service terminals; The obtaining of the first business data written by the first client in the first computer room into the first cache in the first computer room includes: Determine whether the service component identification information corresponding to the first server is the same as the service component identification information carried in the first business data; wherein different first service components correspond to different service component identification information; If so, obtain the first business data written into the first cache of the first computer room by the first client of the first computer room.

7. The method according to claim 6, characterized in that The obtaining of the first business data written by the first client in the first computer room into the first cache in the first computer room includes: If the service component identification information corresponding to the first server is changed, before the change is completed, the first service data written into the first cache is written into the blocking queue; After the change is completed, the first service data is obtained from the blocking queue.

8. The method according to claim 5, characterized in that The first client includes a configuration file, and the configuration file includes a calling address; if the calling address is not empty, the first server is the second type of server.

9. The method according to claim 1, characterized in that: The first service data carries the room identifier of the room that sent the first service data; wherein, If the computer room identifier carried in the first service data is the same as the computer room identifier of the second computer room, the first service data is not written into the second cache; and / or If the computer room identifier carried in the first service data is different from the computer room identifier of the second computer room, the first service data is written into the second cache.

10. The method according to claim 1, characterized in that The method further comprises: Synchronize the first vector clock and the first system clock corresponding to the first service data to the second synchronization component through the first synchronization component; The first vector clock and the first system clock synchronized to the second synchronization component are used by the second server to determine whether to write the first business data synchronized to the second synchronization component into the second cache.

11. The method according to claim 10, characterized in that When any of the following conditions is met, it is determined that the first service data synchronized to the second synchronization component is written into the second cache: The value of the first vector clock is greater than the value of the second vector clock in the second cache; The value of the first vector clock is equal to the value of the second vector clock, and the value of the first system clock is greater than the value of the second system clock in the second cache; The value of the first vector clock is equal to the value of the second vector clock, the value of the first system clock is equal to the value of the second system clock, and the priority of the first computer room is higher than the priority of the second computer room.

12. The method according to claim 10, characterized in that The first vector clock and the first system clock are stored in the first cache in the form of a key-value pair, wherein the field name of the first vector clock is used as the key of the key-value pair, and the value of the first vector clock and the value of the first system clock are used as the value of the key-value pair after hash processing, and each time business data is written to the first cache, the value of the first vector clock is updated according to a preset update method.

13. The method according to claim 1, characterized in that The first computer room further includes a first verification component, and the second computer room further includes a second verification component, and the first verification component is in communication connection with the second verification component; The second verification component can obtain the first business data synchronized to the second synchronization component, and synchronize the obtained first business data to the first verification component; The first verification component can obtain the first business data written into the first cache through the first synchronization component, and perform consistency verification on the first business data synchronized to the second synchronization component based on the first business data obtained from the second verification component and the first business data obtained from the first synchronization component.

14. The method according to claim 13, characterized in that The second verification component subscribes to first service data from a message queue corresponding to the target synchronization link in the second synchronization component based on the target synchronization link between the second computer room and the first computer room, and sends the subscribed first service data to the first verification component in the first computer room; Among them, different synchronization links are data transmission links between different first computer rooms and / or different second computer rooms, and each synchronization link corresponds to a message queue.

15. The method according to claim 13, characterized in that The first business data obtained by the first verification component from the second verification component and the first business data obtained by the first verification component from the first synchronization component are both stored in the etcd of the first computer room, and the first verification component reads them from the etcd of the first computer room and performs consistency verification.

16. The method according to claim 15, characterized in that The consistency check includes: Performing consistency check on a key of the first business data obtained from the second verification component and a key of the first business data obtained from the first synchronization component; The value of the first business data stored in etcd of the first computer room is set to a preset value, and the storage space occupied by the preset value is smaller than the storage space occupied by the actual value of the first business data.

17. The method according to claim 16, characterized in that The consistency check also includes: Based on the source vector clock and the first system clock of the first computer room and the second vector clock and the second system clock of the second computer room, the value of the first business data obtained from the second verification component and the value of the first business data obtained from the first synchronization component are verified.

18. A data synchronization method, characterized in that: A second server end applied to a second computer room, the second server end comprising a second synchronization component, the second synchronization component being communicatively connected to a first synchronization component of a first computer room; the method comprising: Monitor the operation of synchronizing the first business data to the second synchronization component; after the first business data is written into the first cache of the first computer room by the first client of the first computer room, it is acquired by the first server of the first computer room and synchronized to the second synchronization component through the first synchronization component; If detected, the first business data synchronized to the second synchronization component is written into the second cache of the second computer room.

19. The method according to claim 18, characterized in that The first business data carries a computer room identifier of a computer room that sends the first business data; and the step of writing the first business data synchronized to the second synchronization component into the second cache of the second computer room includes: If the computer room identifier carried in the first service data is the same as the computer room identifier of the second computer room, not writing the first service data into the second cache; and / or If the computer room identifier carried in the first service data is different from the computer room identifier of the second computer room, the first service data is written into the second cache.

20. The method according to claim 18, characterized in that The method further comprises: monitoring a first vector clock and a first system clock synchronized to the second synchronization component, where the first vector clock and the first system clock are vector clocks and system clocks corresponding to the first service data; Obtaining a second vector clock and a second system clock in the second cache; Based on the first vector clock, the first system clock, the second vector clock, and the second system clock, it is determined whether to write the first business data synchronized to the second synchronization component into the second cache.

21. The method according to claim 20, characterized in that The second vector clock and the second system clock are stored in the second cache in the form of a key-value pair, wherein the field name of the second vector clock serves as the key of the key-value pair, the value of the second vector clock and the value of the second system clock are hashed as the value of the key-value pair, and the value of the second vector clock is updated according to a preset update method.

22. A data synchronization device, characterized in that: The device is applied to a first server in a first computer room, wherein the first server includes a first synchronization component, and the first synchronization component is communicatively connected with a second synchronization component of a second server in a second computer room; the device includes: An acquisition module, used for acquiring first business data written by a first client in the first computer room into a first cache in the first computer room; A synchronization module, configured to synchronize the first service data to the second synchronization component via the first synchronization component; The first business data synchronized to the second synchronization component can be written into the second cache of the second computer room through the second server, and data is isolated between the first cache and the second cache.

23. A data synchronization device, characterized in that: A second server end applied to a second computer room, wherein the second server end includes a second synchronization component, and the second synchronization component is communicatively connected with the first synchronization component of the first computer room; the device includes: A monitoring module, used for monitoring the operation of synchronizing the first business data to the second synchronization component; after the first business data is written into the first cache of the first computer room by the first client of the first computer room, it is acquired by the first server of the first computer room and synchronized to the second synchronization component through the first synchronization component; The writing module is used to write the first business data synchronized to the second synchronization component into the second cache of the second computer room if monitored.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 21 is implemented.

25. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 21 is implemented.

26. A distributed data processing system, characterized in that: It includes a first server, a first client, and a first cache located in a first computer room, and a second server and a second cache located in a second computer room; the first server includes a first synchronization component, the second server includes a second synchronization component, and the first synchronization component is in communication connection with the second synchronization component; The first client is used to write first service data into the first cache; The first server is used to obtain the first business data written into the first cache, and synchronize the first business data to the second synchronization component through the first synchronization component; The second server is used to write the first business data synchronized to the second synchronization component into the second cache, and the data between the second cache and the first cache is isolated.

Images