Data synchronization method and device, equipment, storage medium and program product

Abstract

The invention provides a data synchronization method and device, equipment, a storage medium and a program product, and relates to the technical field of the Internet, the method comprises the steps that a first command sent by a data synchronization component of a second cluster is received, the first command comprises a Redis instance identifier and a data identifier, and the data identifier indicates a site of synchronized data; the second cluster and the first cluster are deployed in a remote environment; determining target data according to the first command; the target data comprises first incremental data and second incremental data in incremental data written by a user, the first incremental data is data on a site indicated by the data identifier, and the second incremental data is data corresponding to the Redis instance identifier after the first incremental data; and synchronizing the target data to a data synchronization component of the second cluster in an incremental synchronization mode. According to the invention, efficient, low-cost and safe dual-active data synchronization is realized.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data synchronization method, device, equipment, storage medium and program product. Background Art

[0002] In the related art, active-active data synchronization is generally performed by dropping files to disk or by disguising from a processor. However, when performing active-active data synchronization by dropping files to disk, the data needs to be sent to the other end first, and the other end determines whether it is loop data. This takes up a lot of network bandwidth and thread processing resources. In addition, if the network is interrupted, full data synchronization is required, resulting in a very high synchronization cost. The solution of performing active-active data synchronization by disguising from a processor only considers the active-active method, and it is necessary to change the original sync command of the remote dictionary server (redis) itself to add an id. It cannot be adapted to the open source native Redis, which is prone to code modification and intrusion. In addition, solving data consistency through timestamps is risky. If there is a long delay in data synchronization, the reciprocating cycle will cause the system to crash. Therefore, how to achieve efficient, low-cost and safe active-active data synchronization has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a data synchronization method, apparatus, device, storage medium and program product, which solve the problem that the number of SIDs and the complexity of routes are very high in the current method of allocating SIDs for each service and publishing related routes.

[0004] In a first aspect, to achieve the above-mentioned purpose, an embodiment of the present application provides a data synchronization method, which is applied to a master node of a first cluster, and the method includes:

[0005] Receive a first command sent by a data synchronization component of a second cluster, the first command including a remote dictionary service Redis instance identifier and a data identifier, the data identifier indicating a location of synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment;

[0006] According to the first command, the target data is obtained; the target data includes: the first incremental data and the second incremental data in the incremental data written by the user, the first incremental data is the data at the location indicated by the data identifier, and the second incremental data is the data after the first incremental data and corresponding to the Redis instance identifier;

[0007] The target data is synchronized to the data synchronization component of the second cluster in an incremental synchronization manner.

[0008] Wherein, acquiring target data according to the first command includes:

[0009] Search the cache area of ​​the master node to see whether the data corresponding to the data identifier exists. If so, obtain the target data from the cache area. If not, obtain the target data from the distributed storage system corresponding to the master node.

[0010] Wherein, the method further comprises:

[0011] In the case where it is detected that incremental data is written into the buffer of the master node, the incremental data written is collected in an asynchronous manner;

[0012] Generate an identification group corresponding to the incremental data written, wherein the identification group includes at least one of a data shard identification, a data identification, and a database identification; wherein the incremental data written includes one or more incremental data, and each incremental data corresponds to an identification group;

[0013] The collected incremental data and the identification group corresponding to the incremental data are written into a logical log file, and the logical log index file is updated; wherein the logical log file and the logical log index file are stored in a distributed storage system corresponding to the master node, wherein the logical log index file includes the name of the logical log file and the data identification corresponding to the logical log file.

[0014] Wherein, acquiring the target data in the distributed storage system corresponding to the master node includes:

[0015] Searching the logical log index file for a logical log file corresponding to the data identifier;

[0016] In the found logical log file, the target data is retrieved according to the data identifier and the data slice identifier.

[0017] The step of writing the incremental data and the identification group corresponding to the incremental data into a logical log file and updating the logical log index file comprises:

[0018] Loading the logical log index file and the logical log file in the distributed storage system into the memory of the master node;

[0019] Writing the collected incremental data and the identification group corresponding to the incremental data into the logical log file;

[0020] Based on the updated logical log file, updating the logical log index file;

[0021] The updated logical log file and the logical log file index are written into the distributed storage system.

[0022] Wherein, after writing the collected incremental data and the identification group corresponding to the incremental data into the logical log file and updating the logical log index file, the method further comprises:

[0023] Updating a memory structure corresponding to the logical log file, wherein the memory structure includes a handle corresponding to the logical log file and a file offset currently corresponding to the logical log file;

[0024] A notification message of a logical log index file update is sent to a standby node corresponding to the master node of the first cluster, wherein the notification message is used to instruct the standby node to update the logical log index file in the standby node memory based on the logical log index file in the distributed storage system, and to update the memory structure in the standby node memory based on the logical log file in the distributed storage system.

[0025] Among them, the master node and the backup node corresponding to the master node share a distributed storage system, and the master node has write operation permissions and read operation permissions for the logical log files and logical log index files in the distributed storage system, and the backup node has read operation permissions for the logical log files and logical log index files in the distributed storage system.

[0026] Wherein, the method further comprises:

[0027] When the target data is synchronized to the data synchronization component of the second cluster in an incremental synchronization manner, a logical log offset corresponding to the currently synchronized target data is sent to the data synchronization component of the second cluster.

[0028] Before receiving the first command sent by the data synchronization component of the second cluster, the method further includes:

[0029] Receive a second command sent by the data synchronization component of the second cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data;

[0030] In response to the second command, a Redis instance identifier corresponding to the incremental data currently synchronized between the first cluster and the second cluster is sent to the data synchronization component of the second cluster.

[0031] Wherein, the method further comprises:

[0032] Receive the pipeline sending command from the client;

[0033] When the client type carried in the pipeline sending command indicates that the client is a data synchronization component, executing an operation corresponding to the pipeline sending command;

[0034] When the client type carried in the pipeline sending command indicates that the client is not a data synchronization component, the logical log file and the logical log index file are updated according to the pipeline sending command, and the data corresponding to the pipeline sending command is synchronized to clusters other than the first cluster.

[0035] In a second aspect, to achieve the above-mentioned purpose, an embodiment of the present application provides a data synchronization method, which is applied to a data synchronization component of a second cluster, and the method includes:

[0036] In the event that a connection is interrupted during the process of synchronizing incremental data between the data synchronization component of the second cluster and the master node of the first cluster, a first command is sent to the master node of the first cluster, where the first command includes a Redis instance identifier and a data identifier, where the Redis instance identifier is an identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicates the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment;

[0037] Receive target data synchronized by the master node of the first cluster, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier;

[0038] The target data is forwarded to a master node in the second cluster corresponding to the data synchronization component.

[0039] Wherein, the method further comprises:

[0040] When receiving the target data, a logical log offset corresponding to the target data currently synchronized is received.

[0041] Before sending the first command to the master node of the first cluster, the method further includes:

[0042] Sending a second command to the master node of the first cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data;

[0043] Receive the Redis instance identifier sent by the master node of the first cluster.

[0044] The step of forwarding the target data to a master node in the second cluster corresponding to the data synchronization component includes:

[0045] Filtering the data to be transmitted in the target data based on the data shard identifier corresponding to each piece of data in the target data;

[0046] Determine a data channel for transmitting each piece of data to be transmitted according to a database identifier, a data shard identifier, a hash value corresponding to a key value of each piece of data to be transmitted, and a predefined number of data channels N;

[0047] On each of the data channels, the data to be transmitted corresponding to the data channel is asynchronously transmitted in parallel to the master node of the second cluster.

[0048] In a third aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a data synchronization device, which is applied to a master node of a first cluster, and the device includes:

[0049] A first receiving module, configured to receive a first command sent by a data synchronization component of a second cluster, wherein the first command includes a remote dictionary service Redis instance identifier and a data identifier, wherein the data identifier indicates a location of synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment;

[0050] an acquisition module, configured to acquire target data according to the first command; the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data being data at the location indicated by the data identifier, and the second incremental data being data after the first incremental data and corresponding to the Redis instance identifier;

[0051] The first sending module is used to synchronize the target data to the data synchronization component of the second cluster in an incremental synchronization manner.

[0052] In a fourth aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a data synchronization device, which is applied to a data synchronization component of a second cluster, and the device includes:

[0053] A first sending module is used to send a first command to the master node of the first cluster when a connection interruption occurs during the process of synchronizing incremental data between the data synchronization component of the second cluster and the master node of the first cluster, wherein the first command includes a Redis instance identifier and a data identifier, wherein the Redis instance identifier is an identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicates the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment;

[0054] A first receiving module is used to receive target data synchronized by the master node of the first cluster, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier;

[0055] The second sending module is used to forward the target data to the master node corresponding to the data synchronization component in the second cluster.

[0056] In the fifth aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a data synchronization device, including a transceiver, a processor, a memory, and a program stored on the memory and executable on the processor; when the processor executes the program, it implements the data synchronization method as described in the first aspect, or implements the data synchronization method as described in the second aspect.

[0057] In the sixth aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by the processor, the data synchronization method as described in the first aspect is implemented, or the data synchronization method as described in the second aspect is implemented.

[0058] In the seventh aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the data synchronization method as described in the first aspect, or implement the data synchronization method as described in the second aspect.

[0059] The beneficial effects of the above technical solution of the present application are as follows:

[0060] In the data synchronization method of the embodiment of the present application, first, the master node of the first cluster receives the first command sent by the data synchronization component of the second cluster, the first command includes the remote dictionary service Redis instance identifier and the data identifier, and the data identifier indicates the location of the synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment; secondly, according to the first command, the target data is determined; the target data includes: the first incremental data and the second incremental data in the incremental data written by the user, the first incremental data is the data at the location indicated by the data identifier, and the second incremental data is the data after the first incremental data and corresponding to the Redis instance identifier; finally, the target data is synchronized to the data synchronization component of the second cluster in an incremental synchronization manner. In this way, when an abnormality occurs during the data synchronization process and causes the connection to be interrupted (data synchronization interruption), breakpoint resumption can be performed based on the first command, avoiding frequent full data synchronization and the situation where loopback data occupies a large amount of network bandwidth and thread processing resources, and can reduce the phenomenon of code modification and intrusion, and only solve the risk of data consistency through timestamps, and finally achieve efficient, low-cost and safe completion of active-active data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is one of the flowcharts of the data synchronization method according to an embodiment of the present application;

[0062] Figure 2 This is a second flowchart of the data synchronization method according to an embodiment of the present application;

[0063] Figure 3 A schematic diagram of the architecture of a system to which the data synchronization method according to an embodiment of the present application is applied;

[0064] Figure 4 This is a schematic diagram of the collaborative work of the master and backup nodes in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the process of synchronizing data between the master and standby nodes in an embodiment of the present application;

[0066] Figure 6 This is a schematic diagram of the process of redis remote transmission in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of searching for data corresponding to an opid tag in an embodiment of the present application;

[0068] Figure 8 This is one of the schematic diagrams of parallel data transmission in an embodiment of the present application;

[0069] Fig. 9 This is the second schematic diagram of parallel data transmission in an embodiment of the present application;

[0070] Fig.10 This is one of the structural schematic diagrams of the data synchronization device according to an embodiment of the present application;

[0071] Fig.11 This is a second structural diagram of the data synchronization device according to an embodiment of the present application;

[0072] Fig.12 A schematic diagram of the structure of a data synchronization device according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the technical problems, technical solutions and advantages to be solved by the present application clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0074] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0075] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0076] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0077] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, but B can also be determined according to A and / or other information.

[0078] Before describing the embodiments of the present application, the relevant technical points are first described:

[0079] 1. In related technologies, active-active data synchronization uses the following two methods:

[0080] 1) Synchronize active-active data by writing files to disk

[0081] The native data storage method of Redis is Append Only File (AOF). The most common method in the industry is to transform AOF, such as transforming AOF log into data log formats such as logical log (binlog) or write-ahead logging system (WAL), and adding Redis instance ID to the log to solve cyclic synchronization. Among them, each data log contains a globally unique ID (similar to the global transaction identifier (GTID) in MySQL's binlog), where the ID contains two parts: one part is the Redis instance ID, which is used to solve the problem of cyclic synchronization, and the second part is an increasing number to ensure order and uniqueness. The data log contains logical clock information. When Redis on the target side performs a merge, the data synchronization problem is solved by using a conflict-free replication data strategy. This method solves cyclic synchronization and can achieve final data synchronization.

[0082] However, when synchronizing data to the peer end, this method requires first sending the data to the peer end, which then determines whether it is loop data, which takes up a lot of network bandwidth and thread processing resources. In addition, the primary and standby nodes cannot share the storage system, resulting in double write processes. Finally, this part does not segment the files that fall to the disk, nor does it implement breakpoint-resume transmission. If the network is interrupted, the cost of full data synchronization is very high.

[0083] 2) Perform active-active data synchronization by masquerading as a slave processor

[0084] The native data synchronization device of Redis synchronizes data on the standby node by means of the psync command (a command used for master-slave replication in Redis). Therefore, another commonly used method is that the data synchronization tool disguises itself as a slave node for processing, such as set kv, disguising the data synchronization tool as a slave node. Since Redis provides application programming interfaces (APIs) in multiple languages, the data synchronization tool suitable for disguising in this way can be almost any existing general data synchronization tool. The disguised data synchronization tool slave node is responsible for receiving the incremental data sent by the master node, and then encapsulating the data in a new SYNC command (a command for refreshing the file system buffer) in the Redis format. The command will be marked with the ID of the cluster and synchronized to another cluster. After receiving the new SYNC command, the Redis server of another cluster parses the above-mentioned incremental data synchronization data set kv and executes it, so that the cached data is synchronized. At the same time, the Redis server of another cluster pushes the new SYNC command to the data synchronization tool slaves within the cluster. Here, the first cluster is defined as cluster A, and the other cluster is defined as cluster B. The data synchronization tool slave principle of cluster B is the same as that of cluster A, and is also encapsulated by the data synchronization tool through the Redis protocol. After receiving the replication request, the data synchronization tool slave in cluster B parses and judges the request: if it is a new SYNC command, it parses whether its cluster A ID is consistent with the target cluster ID to be synchronized. If they are consistent, it does not need to be synchronized to the target cluster, thus avoiding an infinite loop. At the same time, a timestamp is added to each data to solve the problem of data inconsistency.

[0085] However, this method only considers the active-active method, and needs to change the original sync command of redis itself to add the id, which cannot be adapted to the open source native Redis. The above method synchronizes the command to master A through data synchronization tool B, and then synchronizes master A to data synchronization tool A again. Then, to determine whether to synchronize to master A, it is necessary to change the native redis, resulting in code modification and intrusion. At the same time, solving data consistency through timestamps is risky. If there is a long delay in data synchronization, the cycle will cause system paralysis.

[0086] Based on the above related technologies, the embodiments of the present application provide a data synchronization method, device, equipment, medium and program product. The embodiments of the present application will be described in detail below with reference to the accompanying drawings and specific examples.

[0087] The data synchronization method provided in the embodiment of the present application is applied to the master node of the first cluster, such as Figure 1 As shown, the method includes:

[0088] Step 101, receiving a first command sent by a data synchronization component of a second cluster, wherein the first command includes a remote dictionary service Redis instance identifier and a data identifier, wherein the data identifier indicates a location of synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment;

[0089] In the above steps, first, the data synchronization component is a self-developed multi-directional synchronization component used for data transmission between multiple clusters, such as Figure 3 As shown, the data synchronization components are mslave1, mslave2, mslave3, etc., and each data synchronization component corresponds to a master node. Secondly, the first command is a command sent by the second cluster to the first cluster when the connection between the first cluster and the second cluster is interrupted during the incremental data synchronization process. Based on the data identifier of the first command, the master node of the first cluster can understand the location of the data synchronized with the second cluster, which is convenient for subsequent breakpoint resumption and avoids frequent full data synchronization; based on the Redis instance identifier of the first command, the master node of the first cluster can understand the Redis instance corresponding to the master node of the second cluster, so as to continue to transmit the data of the Redis instance to the second cluster, avoiding the occurrence of loop data occupying a large amount of network bandwidth and thread processing resources; in addition, the subsequent data synchronization based on the first command can also reduce the situation of code modification and intrusion, as well as the risk of achieving data consistency only through timestamps; thirdly, the first cluster and the second cluster are deployed in a remote environment, which means that the first cluster and the second cluster are deployed in container orchestration engine (kubernetes, K8S) environments in different availability zones, such as the first cluster is deployed in a K8S environment in a first geographical location (such as Beijing), and the second cluster is deployed in a K8S environment in a second geographical location (such as Shenzhen).

[0090] Step 102: determine target data according to the first command; the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier;

[0091] Here, it should be noted that, during the data synchronization between the first cluster and the second cluster (including full data synchronization after the first cluster and the second cluster initially establish a dual-active system link and / or incremental data synchronization between the first cluster and the second cluster), the user will continue to write incremental data to the first cluster, and the above step 102 is used to determine the incremental data that has not been synchronized with the second cluster from the incremental data continuously written by the user;

[0092] Step 103: synchronize the target data to the data synchronization component of the second cluster in an incremental synchronization manner.

[0093] In the data synchronization method of the embodiment of the present application, first, the master node of the first cluster receives the first command sent by the data synchronization component of the second cluster, the first command includes the remote dictionary service Redis instance identifier and the data identifier, and the data identifier indicates the location of the synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment; secondly, according to the first command, the target data is determined; the target data includes: the first incremental data and the second incremental data in the incremental data written by the user, the first incremental data is the data at the location indicated by the data identifier, and the second incremental data is the data after the first incremental data and corresponding to the Redis instance identifier; finally, the target data is synchronized to the data synchronization component of the second cluster in an incremental synchronization manner. In this way, when an abnormality occurs during the data synchronization process and causes the connection to be interrupted (data synchronization interruption), breakpoint resumption can be performed based on the first command, avoiding frequent full data synchronization and the situation where loopback data occupies a large amount of network bandwidth and thread processing resources, and can reduce the phenomenon of code modification and intrusion, and only solve the risk of data consistency through timestamps, and finally achieve efficient, low-cost and safe completion of active-active data synchronization.

[0094] As an optional implementation, step 102 includes:

[0095] Search the cache area of ​​the master node to see whether the data corresponding to the data identifier exists. If so, obtain the target data from the cache area. If not, obtain the target data from the distributed storage system corresponding to the master node.

[0096] That is to say, after the master node of the first cluster obtains the first command sent by the data synchronization component of the second cluster, it further searches its cache area for data corresponding to the data identifier in the first command. If so, it indicates that the data that the first cluster needs to synchronize to the second cluster in the future is stored in the cache area of ​​the master node of the first cluster, so the target data can be obtained from the cache area. If not, it indicates that the data that the first cluster needs to synchronize to the second cluster in the future has been written into the distributed storage system corresponding to the master node, so it is necessary to further search the distributed storage system for data corresponding to the data identifier in the first command. If so, it indicates that at least part of the data that the first cluster needs to synchronize to the second cluster in the future is stored in the distributed storage system corresponding to the master node of the first cluster, so the target data can be obtained from the distributed storage system.

[0097] Furthermore, as an optional implementation, the method further includes:

[0098] (1) When it is detected that incremental data is written into the buffer of the master node, the incremental data written is collected in an asynchronous manner; Figure 4 For example, in this step, the collector in the master node asynchronously collects the incremental data.

[0099] (2) generating an identification group corresponding to the incremental data to be written, wherein the identification group includes at least one of a data shard identification, a data identification, and a database identification; wherein the incremental data to be written includes one or more incremental data, and each incremental data corresponds to an identification group; that is, for each incremental data, a corresponding identification group is generated. As an example, the format of each data can be expressed as:

[0100] #db:db0;shardId:shardid0;opid:opid0;*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nv alue\r\n

[0101] Among them, db represents the database identifier corresponding to the data, shardid represents the data shard identifier (also called redis cluster identifier) ​​corresponding to the data, and opid represents the data identifier of the data; among them, opid definition includes incNumber and time. incNumber is an auto-increment integer, and time represents the current timestamp at the microsecond level. Specifically, opid = time*10 10 +incNumber.

[0102] (3) writing the collected incremental data and the identification group corresponding to the incremental data into a logical log file, and updating a logical log index file; wherein the logical log file and the logical log index file are stored in a distributed storage system corresponding to the master node, wherein the logical log index file includes the name of the logical log file and the data identification corresponding to the logical log file.

[0103] In the above optional implementation, incremental data is collected asynchronously and an identification group corresponding to each incremental data is generated, so that the incremental data and the corresponding identification group are stored in a distributed storage system, which facilitates subsequent search for the corresponding incremental data in the distributed storage system based on the identification group.

[0104] As a specific implementation manner, obtaining the target data in the distributed storage system corresponding to the master node includes:

[0105] Searching the logical log file corresponding to the data identifier in the logical log index file; as an example, this step can use a binary search algorithm to search the logical log file corresponding to the data identifier in the logical log index file. The binary search algorithm is also called a binary search algorithm, which is a highly efficient search algorithm.

[0106] In the found logical log file, the target data is retrieved according to the data identifier and the data shard identifier. As an example, this step may be to sequentially retrieve the data where the data identifier is located / corresponds to from the offset0 position of the logical log file, thereby reducing the number of file scan lines and thus reducing the disk access time.

[0107] As another specific implementation, writing the incremental data and the identification group corresponding to the incremental data into a logical log file, and updating the logical log index file, includes:

[0108] Loading the logical log index file and the logical log file in the distributed storage system into the memory of the master node;

[0109] Writing the collected incremental data and the identification group corresponding to the incremental data into the logical log file;

[0110] Based on the updated logical log file, the logical log index file is updated; here, since the logical log index file includes the name of the logical log file and the data identifier corresponding to the logical log file (the data identifier in the incremental data included in the logical log file), after the collected incremental data and its corresponding identification group are written in the logical log file, it is necessary to update the logical log index file based on the written identification group and the name of the logical log file, specifically, write the corresponding relationship between the data identifier of the incremental data newly added to the logical log file and the name of the logical log file in the logical log index file;

[0111] Here, it should be noted that since the size of the logical log file will affect the disk writing efficiency, the logical log file is generally limited to 1G. Therefore, when writing data to the logical log file, the logical log file will be split. Therefore, there will be multiple logical log files in the distributed storage system. Therefore, the distributed storage system also needs a logical log index file to facilitate the rapid retrieval of logical log files.

[0112] The updated logical log file and the logical log file index are written into the distributed storage system.

[0113] Further, as an optional implementation, after writing the collected incremental data and the identification group corresponding to the incremental data into a logical log file and updating the logical log index file, the method further includes:

[0114] Updating a memory structure corresponding to the logical log file, wherein the memory structure includes a handle corresponding to the logical log file and a file offset currently corresponding to the logical log file;

[0115] A notification message of a logical log index file update is sent to a standby node corresponding to the master node of the first cluster, wherein the notification message is used to instruct the standby node to update the logical log index file in the standby node memory based on the logical log index file in the distributed storage system, and to update the memory structure in the standby node memory based on the logical log file in the distributed storage system.

[0116] In the above optional implementation, after the logical log file and the logical log index file in the distributed storage system are updated, the memory structure is further updated and a notification message is sent to the standby node corresponding to the master node, so that the data of the master node and the standby node can be consistent. Therefore, when the master and standby nodes switch due to abnormal conditions, the standby node can start working in time without affecting the normal business services of the cluster.

[0117] As a specific implementation method, the master node and the backup node corresponding to the master node share a distributed storage system, and the master node has write operation permissions and read operation permissions for the logical log files and logical log index files in the distributed storage system, and the backup node has read operation permissions for the logical log files and logical log index files in the distributed storage system.

[0118] In the above specific implementation method, the master node and the backup node share a distributed storage system, which can reduce the number of disk operations and greatly improve the efficiency of disk placement. It can solve the problem that the existing master and backup nodes each have a distributed storage system with large write traffic, which affects performance. Among them, the master node is set to have write operation permissions and read operation permissions, and the backup node is only set to read operation permissions, which can ensure data consistency when concurrently reading and writing files or switching between master and backup nodes.

[0119] As an optional implementation, the method further includes:

[0120] When synchronizing the target data to the data synchronization component of the second cluster in an incremental synchronization manner, the logical log offset corresponding to the currently synchronized target data is sent to the data synchronization component of the second cluster. Here, when synchronizing incremental data, the logical log offset is also synchronized to perform disk access based on the logical log offset, thereby speeding up access speed and improving access efficiency.

[0121] As an optional implementation, before step 101, the method further includes:

[0122] Receive a second command sent by the data synchronization component of the second cluster, where the second command is used to request the Redis instance identifier corresponding to the currently synchronized incremental data; here, the second command is, for example, a dask command; the Redis instance identifier can be Figure 6 The clusterid in represents;

[0123] In response to the second command, a Redis instance identifier corresponding to the incremental data currently synchronized between the first cluster and the second cluster is sent to the data synchronization component of the second cluster.

[0124] In the above optional implementation, the data synchronization component of the second cluster sends a second command to obtain the Redis instance identifier corresponding to the incremental data synchronized between the two clusters fed back by the master node of the first cluster, so that when the connection is interrupted, the data synchronization component of the second cluster sends the aforementioned first command to the master node of the first cluster to achieve breakpoint resumption of incremental data.

[0125] Furthermore, as an optional implementation, the method further includes:

[0126] Receive the command sent by the client pipeline;

[0127] When the client type carried in the pipeline send command indicates that the client is a data synchronization component, the operation corresponding to the pipeline send command is executed; here, first, for example, when the client type is the DTS type, it indicates that the client is a data synchronization component; second, the operation corresponding to the pipeline send command is, for example, updating the data state of the data corresponding to the pipeline send command; third, when the client is a data synchronization component, it indicates that the data corresponding to the pipeline send command sent by the client is not the data generated by the local instance and does not need to be written to the local disk, so only the pipeline send command needs to be executed;

[0128] In the case where the client type carried in the pipeline sending command indicates that the client is not a data synchronization component, the logical log file and the logical log index file are updated according to the pipeline sending command, the received data is logically written to the disk, and the data corresponding to the pipeline sending command is synchronized to clusters outside the first cluster. Here, when the client is not a data synchronization component but a normal client, it indicates that the data corresponding to the pipeline sending command sent by the client is data generated by the local instance, and these data need to be written to the disk. Therefore, it is necessary to update the logical log file, the logical log index file, and the memory structure corresponding to the logical log file based on these incremental data to achieve the writing of incremental data to the disk.

[0129] Based on the above optional implementations, the embodiments of the present application also include:

[0130] After the master node of the first cluster establishes a persistent connection with the corresponding data synchronization component, the type of the persistent connection is set to DTS / DTS_client, and the master node saves the type to the replication type connection and adds the role_sub:dts field (to distinguish ordinary replication connections). In this way, after receiving a pipeline concurrent command, it can be determined whether the client sending the pipeline concurrent command is a data synchronization component based on the above settings.

[0131] The embodiment of the present application also provides a data synchronization method, which is applied to the data synchronization component of the second cluster, such as Figure 2 As shown, the method includes:

[0132] Step 201: When a connection interruption occurs during the process of synchronizing incremental data between the data synchronization component of the second cluster and the master node of the first cluster, a first command is sent to the master node of the first cluster, where the first command includes a Redis instance identifier and a data identifier, where the Redis instance identifier is an identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicates the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment;

[0133] Here, it should be noted that the Redis instance identifier in the above steps is the Redis instance corresponding to the data to be pulled. Figure 6 For example, the Redis instance identifier can be expressed as clusterid;

[0134] Step 202: Receive target data synchronized by the master node of the first cluster, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier;

[0135] Step 203: forward the target data to the master node corresponding to the data synchronization component in the second cluster.

[0136] In an embodiment of the present application, first, when a connection interruption occurs during the process of incremental data synchronization between the data synchronization component of the second cluster and the master node of the first cluster, a first command is sent to the master node of the first cluster, the first command including a Redis instance identifier and a data identifier, the Redis instance identifier being the identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicating the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment; secondly, target data synchronized by the master node of the first cluster is received, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data being the data at the location indicated by the data identifier, and the second incremental data being the data after the first incremental data and corresponding to the Redis instance identifier; thirdly, the target data is forwarded to the master node corresponding to the data synchronization component in the second cluster. In this way, when the connection is interrupted during the incremental data synchronization process, data can be resumed based on the Redis instance identifier and data identifier, avoiding frequent full data synchronization; setting the corresponding Redis instance identifier and data identifier in the incremental data can reduce the phenomenon of code modification and intrusion, and can also solve the risks of solving data consistency only through timestamps, and ultimately achieve efficient, low-cost and secure completion of active-active data synchronization.

[0137] Furthermore, as an optional implementation, the method further includes:

[0138] When receiving the target data, the logical log offset corresponding to the target data currently synchronized is received. Here, when synchronizing incremental data, the logical log offset is also synchronized to perform disk access based on the logical log offset, thereby speeding up access speed and improving access efficiency.

[0139] Further, as another optional implementation, before step 201, sending the first command to the master node of the first cluster, the method further includes:

[0140] Sending a second command to the master node of the first cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data; here, the second command is, for example, a dask command;

[0141] Receive the Redis instance identifier sent by the master node of the first cluster.

[0142] In the above optional implementation, the data synchronization component of the second cluster sends a second command to obtain the Redis instance identifier corresponding to the incremental data synchronized between the two clusters fed back by the master node of the first cluster, so that when the connection is interrupted, the data synchronization component of the second cluster sends the aforementioned first command to the master node of the first cluster to achieve breakpoint resumption of incremental data.

[0143] As an optional implementation, forwarding the target data to a master node corresponding to the data synchronization component in the second cluster includes:

[0144] (1) Based on the data shard identifiers corresponding to each piece of data in the target data, the data to be transmitted is screened in the target data; wherein the specific screening method of this step may be: extracting the data shard identifier in the target data, comparing the data shard identifier with the local Redis instance identifier, if the two are the same, discarding the data, if the two are different, determining the data as the data to be transmitted. In this way, the problem of circular replication can be solved.

[0145] (2) determining a data channel for transmitting each piece of data to be transmitted according to the database identifier, the data shard identifier and the hash value corresponding to the key value of each piece of data to be transmitted, and a predefined number of data channels N;

[0146] Here, it should be noted that the above steps specifically segment each received data to achieve the transmission of each segmented sub-data in different data channels. Among them, since the redis source aof file is sequential, the data cannot be segmented arbitrarily, so the data is first transformed so that performance can be accelerated by data segmentation. The data definition is as follows:

[0147] #db:db0;shardId:shardid0;*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n

[0148] Since shardid represents different redis instances, clustering data according to shardid will not affect data consistency; db represents the redis database number, and clustering data with different database numbers will not affect data consistency; finally, the transmission of different key values ​​will not affect data consistency; therefore, the data segmentation formula is defined as follows:

[0149] channelid=(shardid*10 10 +db*10 5 +keyslot)%N

[0150] Among them, keyslot represents the hash value corresponding to the key value, occupying 5 bits; db represents the db number corresponding to the key value, occupying 5 bits; shardid represents the redis data shard number; N represents the definition of the data channel; channelid represents the corresponding channel number after the data is segmented.

[0151] In addition, since redis has multiple command operations, such as mset k1 v1 k2 v2, this type of command cannot be directly applied to the above formula for data segmentation. For this type of command, the command can be segmented and then calculated separately; for example, the above command is split into (mset k1 v1; mset k2 v2; mset k3 v3).

[0152] (3) On each of the data channels, asynchronously and in parallel transmit the to-be-transmitted data corresponding to the data channel to the master node of the second cluster.

[0153] By adopting the processing logic of the above optional implementation method, mslave fully analyzes the data and establishes a concurrent transmission mechanism, which greatly increases the efficiency of data synchronization and makes up for the two major shortcomings of the remote network: time delay and the remote network is not friendly to high concurrency.

[0154] The above-mentioned data synchronization method of the embodiment of the present application proposes a cross-resource pool Redis remote active-active system, and originally proposes a new binlog data persistence method that supports command unique identification, which is different from RDB and AOF, and proposes a method and system for distributed shared storage of redis cluster master and standby nodes, so as to achieve efficient data sharing and loop replication, and finally achieve the ability of two-way transmission between multiple Redis clusters in different regions. Among them, the master and standby nodes share the distributed storage system protocol, support the concurrent read and write access of the master and standby nodes to the binlog files and ensure data consistency in the case of master-standby switching, and effectively support the improvement of read and write efficiency. Among them, the self-developed Redis data synchronization component mslave can optimize the transmission performance by means of data segmentation and establishing a concurrent transmission mechanism, greatly increase the efficiency of data synchronization, make up for the two major shortcomings of the remote network delay and the remote network unfriendly to high concurrency, and at the same time, can ensure the final consistency of the data. In addition, a new redis remote transmission protocol that supports breakpoint resumption is proposed. Specifically, a new long communication connection is constructed, and the master node actively pushes data to the mslave node. When the network is disconnected, the mslave starts to pull the latest data through the latest opid received instead of synchronizing the entire data, thereby optimizing the performance of data transmission across resource pools.

[0155] Specifically, firstly, by designing a new output protocol to realize cross-regional data transmission and replication functions, the effect of two-way synchronization can be achieved, which can overcome the problem that the redis psync replication protocol frequently synchronizes the entire amount in the case of network interruption and network jitter, resulting in low transmission performance or even unavailability. Secondly, since the dual-active data synchronization method is performed by disguising from the processor, only the dual-active synchronization method is supported. If multi-directional synchronization is to be performed, the pressure on the Redis master will be very large. The real-time multi-directional synchronization tool proposed in this application will leave the synchronization to the tool, which will not cause too much burden on Redis itself. Thirdly, in a cross-resource network environment, the data transmission speed is unstable, and the transmission component is prone to backlog of data, resulting in large transmission delays. This application fully analyzes the data and establishes a concurrent transmission mechanism, which greatly increases the efficiency of data synchronization and makes up for the two major shortcomings of the remote network delay and the unfriendliness of the remote network to high concurrency. Fourthly, the native data persistence method of redis cannot achieve two-way synchronization, and the aof rewrite mechanism will cause a sudden increase in the pressure on the central processor of the master node. This application proposes a new binlog persistence solution for redis, which uses distributed storage for master-slave sharing, reduces the number of disk operations, and greatly improves the efficiency of disk placement.

[0156] Here, it should be noted that the system architecture for implementing the above-mentioned remote active-active data synchronization method of the embodiment of the present application includes multiple redis clusters deployed in multiple remote K8S environments. Figure 3 As shown in the figure, taking two availability zones as an example, a redis cluster with the same specifications is deployed in region 1 and region 2, for example, in Beijing and Shanghai. The K8S clusters communicate with each other through remote communication channels ( Figure 3 The cross-resource pool data channel in the rediscluster realizes network communication. When individual shards fail in the rediscluster, the high availability mechanism of redis itself is used to switch between the master and slave. The redis master / slave share distributed storage for storage file sharing. Figure 3 MSlave is a self-developed multi-directional synchronization component. As a data transmission component between multiple clusters, it ensures data synchronization between clusters. Specifically, each master corresponds to one MSlave.

[0157] The implementation details of the above embodiment are respectively described in three aspects, namely, the redis cluster dual-master system, the redis remote transmission protocol, and the mslave data synchronization system.

[0158] (I) Redis cluster dual-master system

[0159] The redis cluster dual-master system adopts the redis cluster cluster mode, in which the master and standby nodes are designed as a group of master and standby nodes that need to work together and share a distributed storage system. The distributed storage system is used to store binlog.index and binlog series files. The redis master and standby nodes are nodes customized for the open source redis, mainly adding three parts: loader, collector, and binlog series file sharing protocol. The main focus of the master and standby node switching is to explain how to ensure the data consistency of binlog in this scenario. The above three parts work together to avoid data loop replication and record the data records that need to be transmitted. The architecture diagram of the redis cluster dual-master system is shown below. Figure 4 shown.

[0160] The Redis master and backup services form a cluster, which is uniquely identified by clusterid. This field indicates the unique id corresponding to the master and backup clusters. When the data contains shardid0, it means that the data is generated by shardid0. A collector will be installed in the master node. The collector will asynchronously collect data in the replication backlog buffer and write it to the binlog file. The loader will load the binlog file in certain scenarios (such as master node abnormality, network timeout, or master and backup node switching scenarios) and send it to the service that needs to transmit data through the network ( Figure 3 The data transmission component corresponding to the master node in the slave node). The collector and loader in the slave node are not working. When the slave node switches to the master node, they will start working.

[0161] (1) Collector: When redis triggers the modification of the replication backlog buffer, the collector collects the data written to the replication backlog buffer. The collector generates three fields: clusterid, opid, and db. The data is written to the binlog file in an asynchronous mode and recorded through the # annotation. The format of each data collected by the collector is as follows:

[0162] #db:db0;shardId:shardid0;opid:opid0;*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nv alue\r\n

[0163] Among them, db represents the database number corresponding to the data (corresponding to the aforementioned database identifier), shardid represents the redis cluster corresponding to the data (corresponding to the aforementioned data shard identifier), and opid is the unique number of the data (corresponding to the aforementioned data identifier). Specifically, the opid definition includes two parts: incNumber and time, where incNumber is an auto-incrementing integer and time represents the current timestamp at the microsecond level, for example: opid = time*10 10 +incNumber;

[0164] (2) Loader: The loader is mainly responsible for loading binlog.index and binlog into memory to facilitate real-time data writing or primary and standby data synchronization.

[0165] (3) Distributed storage system: This part mainly stores two types of files: binlog.index and binlog series. Among them, binlog.index is an index file used to manage multiple binlog series files. The binlog file records the data files that need to be transmitted. Specifically, the binlog file format is as follows:

[0166] #db:db0;shardId:shardid0;opid:opid0;*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nv alue\r\n

[0167] #db:db0;shardId:shardid0;opid:opid1;*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nv alue\r\n

[0168] The format of the Binlog.index file is as follows: it contains multiple binlog files divided by day, and the starting opid number of the binlog:

[0169] File binlog-20240317-01.aof opid 17107543910000000000001

[0170] File binlog-20240319-01.aof opid 17108407910000000000001

[0171] File binlog-20240321-01.aof opid 17110135910000000000001

[0172] (4) Binlog file sharing protocol: Since binlog files are shared through a distributed storage system, special design is required to ensure data consistency in the case of concurrent reading and writing and master-slave switching. In order to achieve consistency in concurrent reading and writing files, the master node can be responsible for writing binlog.index and binlog files, and the slave node has only read permissions.

[0173] In order to achieve data consistency in the memory of the master and standby nodes, a long connection can be established between the master and standby nodes, named as a communicator (such as Figure 5 As shown in Figure 1, this connection is mainly used to synchronize binlog status change information. Specifically, the gossip protocol is used to implement message communication. The gossip protocol is used to spread information to the entire network in a random and contagious manner, and to make all node data in the system consistent within a certain period of time. The specific process of data synchronization between the master and standby nodes is as follows: Figure 5 As shown:

[0174] 1) Update binlog.index file (corresponding to Figure 5 Steps ① and ② in the above example): binlog.index stores the binlog file name and the corresponding opid information. When the binlog file is larger than 1G, the file will be split to improve efficiency and a new binlog file will be created. At this time, the binlog.index memory structure will be updated in real time, and the corresponding disk file binlog.index will be refreshed at the same time.

[0175] 2) The master node updates the memory structure corresponding to the binlog, where cur_binlog_fd represents the handle corresponding to the current binlog file, and cur_binlog_offset represents the file offset corresponding to the current binlog file.

[0176] 3) The master node informs the slave node of the change in binlog.index through the communicator; after receiving the message, the slave node learns that the binlog.index file has been updated, and then loads the latest information into the memory through the loader to ensure that it is consistent with the data of the master node (corresponding to Figure 5 ③ and ④ in the above).

[0177] 4) While updating binlog.index, the slave node uses the binlog file to update the cur_binlog_fd and cur_binlog_offset memory structures in memory to keep them consistent with the master (corresponding to Figure 5 and ⑥ in the above).

[0178] The above operations ensure that the memory information of the active and standby nodes is consistent. In this way, when the active and standby nodes switch, the normal business services of the cluster will not be affected.

[0179] (II) Redis remote transmission protocol

[0180] Remote network communication between multiple K8S Redis clusters is achieved through communication pipelines, but remote communication has problems such as high latency and unstable network. In such a network environment, the native redis psync command will frequently synchronize the entire volume, increase network transmission overhead, and even become unusable. In the embodiment of the present application, a new communication solution that supports breakpoint resumption is proposed to optimize this problem. The specific idea is to build a new long communication connection. The master node actively pushes data to the mslave node (in the remote cluster). When the network is disconnected, the mslave starts to pull the latest data through the latest opid received, instead of performing full data synchronization. The specific implementation is as follows: Figure 6 As shown in the figure, the protocol communication is divided into three parts: full phase, incremental acquisition phase, and new protocol incremental phase.

[0181] Full-scale stage: When the active-active system link is first established (A, B), the A end (by Figure 6 All data of the redis master in the table needs to be synchronized to the B side (represented by Figure 6 mslave in the table), so the sync command can be used for full synchronization. A sends all the data to the remote mslave in RDB (Redis DateBase) mode. RDB mode is a persistence mode of redis database, which mainly stores the data in the memory into a binary file on the hard disk through snapshots.

[0182] Incremental acquisition of site phase: During the RDB data synchronization phase, the user's newly added data is saved in the buffer memory to ensure that the user's newly added data and the RDB file data can represent the full data; when the RDB is sent, the master node will push the user's newly added command data buffer to the mslave node in real time through a long connection; after receiving the data, the mslave finds the first data containing the opid tag (assuming it is opid0), which is the first data that needs to be synchronized after the full RDB data.

[0183] Incremental phase of the new protocol: Transmission starts from opid0 data, which can ensure that no data in the RDB file is transmitted, and that no user-added data is missed, thus ensuring the integrity of the transmitted data.

[0184] In the subsequent process, if the master-slave connection is temporarily interrupted due to network reasons or the network delay causes the psync cache to overflow, the open source redis will retransmit the entire RDB; while this stage in this application adopts a new protocol for incremental transmission, and the breakpoint resume capability can avoid retransmitting the entire RDB, and only transmit the untransmitted part of the data (corresponding to Figure 6 The specific method is as follows:

[0185] 1) Get the peer clusterid through the dask command; corresponding to Figure 6 The mslave in the cluster sends dask?clusterid to the master, and the master feeds back clusterid=clusterid0 to the maslave;

[0186] 2) Define a long connection, and the slave sends a dsync command to the master; the specific form of the dsync command can be: dsync? opid=opid1,clusterid=clusterid0, where opid1 represents the starting number of the data to be retrieved, offset0 represents the starting offset of the file where the data is located, and clusterid0 represents pulling the data corresponding to the redis instance. Therefore, the above dsync command means starting from opid1 (the value obtained in the previous step), sending all the data that the master node needs to synchronize in an incremental push manner, and the offset of binlogoffset needs to be given to the mslave node in the return value to speed up access.

[0187] 3) Check whether the buffer in the memory contains the opid1 data. If it does, send the data in the buffer directly to the mslave;

[0188] If the data of opid1 does not exist in the buffer, the specific binlog file name corresponding to opid1 is searched in the binlog.index file. This part can be searched using a binary search algorithm. Specifically, the data where opid1 is located is sequentially retrieved from the offset0 position of the file, and the remaining data in the binlog is queried in real time from this position and pushed to the peer service (the peer mslave). Among them, the use of file offsets can reduce the number of file scan lines, thereby reducing the time to access the disk.

[0189] 4) After receiving the data synchronized by the master, the mslave filters it through clusterid0 and does not transmit data other than clusterid0.

[0190] 5) Use the long connection to push the data (buffer) generated by subsequent users to the mslave node in real time to perform incremental synchronization. The operation example is as follows: Figure 7 shown.

[0191] From the above process, we can see that the remote transmission protocol proposed this time is perfectly compatible with redis, and uses binlog files and opid to achieve the ability of breakpoint resumption, avoiding the low efficiency of native redis remote transmission and the problem of full RDB retransmission.

[0192] (III) Redis data synchronization component mslave performs real-time synchronization

[0193] In order to ensure data consistency, traditional data synchronization must be transmitted serially in the order in which data is sent. The technology used is to transmit data through psync or a single-threaded (pipeline) pipeline. This transmission mode has poor performance in environments where there is a delay across resource pool networks or where the query rate per second (QPS) is not friendly. This section designs a self-developed Redis data synchronization component mslave, and proposes a method to segment data and establish a concurrent transmission mechanism to optimize transmission performance; Figure 8 and Fig. 9 As shown, the entire component is divided into four parts: data filtering, data segmentation, parallel sending, and connection mode.

[0194] Data filtering: In a dual-active system, when different areas synchronize data, there will be a problem of circular replication. To address this problem, MSlave will first perform data filtering. The filtering method is to extract the label shardid in the command and compare it with the label of the local Redis instance. If they are the same, the data will be discarded. If they are different, the data will be transmitted, thus solving the problem of circular replication.

[0195] Data segmentation: First, the data needs to be segmented. Since the redis source AOF file is sequential, the data cannot be segmented at will. Therefore, the data is first transformed so that performance can be accelerated by data segmentation. The data definition is as follows:

[0196] #db:db0;shardId:shardid0;*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n

[0197] Since shardid represents different redis instances, clustering data according to shardid will not affect data consistency; db represents the redis database number, and clustering data with different database numbers will not affect data consistency; finally, the transmission of different keys will not affect data consistency; therefore, the data segmentation formula is defined as follows:

[0198] channelid=(shardid*10 10 +db*10 5 +keyslot)%N

[0199] Among them, keyslot represents the hash value corresponding to the key value, occupying 5 bits; db represents the db number corresponding to the key value, occupying 5 bits; shardid represents the redis data shard number; N represents the definition of the data channel; channelid represents the corresponding channel number after the data is segmented.

[0200] Redis is operated by multiple commands, such as mset k1 v1 k2 v2. This type of command cannot be directly applied to the above formula for data segmentation. For this type of command, the command can be segmented and then calculated separately. For example, the above command is split into (mset k1 v1; mset k2 v2; mset k3 v3).

[0201] Parallel sending: The data has been divided into parts that can be sent in parallel. This time, N concurrent threads are defined. Each thread starts a client connection as a data sender and sends the data asynchronously to the target redis node through a pipeline.

[0202] Connection mode: Figure 8 As shown in the figure, the data transmitted by mslave is not generated by the local instance and does not need to be written to the local disk; while the data generated by the user connection is a local connection and needs to be written to the disk for transmission; to solve this problem, Fig. 9 As shown, a new client type DTS type is proposed, which means that the mslave synchronization component does not need to store data on the disk or transmit data again. The specific method is:

[0203] 1) Establish a new connection type: As we all know, the client connection type normally used by users is CLIENT_TYPE_NORMAL. All data of this type needs to be stored on disk and transmitted. Here, a new persistent connection is designed and the type of the persistent connection is set to dts_client to distinguish it from ordinary connections.

[0204] 2) The server saves the dts_client type client separately. Here, the dts_client type connection is saved in the replication type connection, and the role_sub:dts field is added to distinguish it from the ordinary replicaiton connection. Fig. 9 shown.

[0205] 3) In the disk write logic, if the client type is found to be dts_client, the binlog will not be written to the disk, but the corresponding command will be executed.

[0206] Through the processing logic of the above four parts, mslave fully analyzes the data and establishes a concurrent transmission mechanism, which greatly increases the efficiency of data synchronization and makes up for the two major shortcomings of the remote network: time delay and the unfriendliness of the remote network to high concurrency.

[0207] The embodiment of the present application also provides a data synchronization device, which is applied to the master node of the first cluster, such as Fig.10 As shown, the device comprises:

[0208] The first receiving module 1001 is used to receive a first command sent by a data synchronization component of a second cluster, wherein the first command includes a remote dictionary service Redis instance identifier and a data identifier, wherein the data identifier indicates a location of synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment;

[0209] The acquisition module 1002 is used to acquire target data according to the first command; the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is the data at the location indicated by the data identifier, and the second incremental data is the data after the first incremental data and corresponding to the Redis instance identifier;

[0210] The first sending module 1003 is configured to synchronize the target data to the data synchronization component of the second cluster in an incremental synchronization manner.

[0211] Optionally, the acquisition module 1002 includes:

[0212] The acquisition submodule is used to search whether the data corresponding to the data identifier exists in the cache area of ​​the master node. If so, the target data is acquired in the cache area; if not, the target data is acquired in the distributed storage system corresponding to the master node.

[0213] Optionally, the device further comprises:

[0214] A collection module, configured to collect the written incremental data in an asynchronous manner when detecting that incremental data is written into the buffer of the master node;

[0215] A generation module, used to generate an identification group corresponding to the incremental data written, wherein the identification group includes at least one of a data shard identification, a data identification, and a database identification; wherein the incremental data written includes one or more incremental data, and each incremental data corresponds to an identification group;

[0216] The first update module is used to write the collected incremental data and the identification group corresponding to the incremental data into a logical log file, and update the logical log index file; wherein the logical log file and the logical log index file are stored in a distributed storage system corresponding to the master node, wherein the logical log index file includes the name of the logical log file and the data identification corresponding to the logical log file.

[0217] Optionally, when the acquisition submodule is used to acquire the target data in the distributed storage system corresponding to the master node, it is specifically used to:

[0218] Searching the logical log index file for a logical log file corresponding to the data identifier;

[0219] In the found logical log file, the target data is retrieved according to the data identifier and the data slice identifier.

[0220] Optionally, the first updating module includes:

[0221] A loading submodule, used to load the logical log index file and the logical log file in the distributed storage system into the memory of the master node;

[0222] A first updating submodule, configured to write the collected incremental data and the identification group corresponding to the incremental data into the logical log file;

[0223] A second updating submodule, configured to update the logical log index file based on the updated logical log file;

[0224] The writing submodule is used to write the updated logical log file and the logical log file index into the distributed storage system.

[0225] Optionally, the device further comprises:

[0226] A second updating module is used to update a memory structure corresponding to the logical log file, wherein the memory structure includes a handle corresponding to the logical log file and a file offset currently corresponding to the logical log file;

[0227] The second sending module is used to send a notification message of logical log index file update to the standby node corresponding to the master node of the first cluster, wherein the notification message is used to instruct the standby node to update the logical log index file in the standby node memory based on the logical log index file in the distributed storage system, and to update the memory structure in the standby node memory based on the logical log file in the distributed storage system.

[0228] Optionally, the master node and the backup node corresponding to the master node share a distributed storage system, and the master node has write operation permissions and read operation permissions for the logical log files and logical log index files in the distributed storage system, and the backup node has read operation permissions for the logical log files and logical log index files in the distributed storage system.

[0229] Optionally, when the first sending module 1003 is used to synchronize the target data to the data synchronization component of the second cluster in an incremental synchronization manner, it is also used to send a logical log offset corresponding to the currently synchronized target data to the data synchronization component of the second cluster.

[0230] Optionally, the device further comprises:

[0231] A second receiving module, used to receive a second command sent by the data synchronization component of the second cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data;

[0232] A third sending module is used to send, in response to the second command, a Redis instance identifier corresponding to the incremental data currently synchronized between the first cluster and the second cluster to the data synchronization component of the second cluster.

[0233] Optionally, the device further comprises:

[0234] The third receiving module is used to receive the pipeline sending command from the client;

[0235] an execution module, configured to execute an operation corresponding to the pipeline sending command when the client type carried in the pipeline sending command indicates that the client is a data synchronization component;

[0236] A processing module is used to update the logical log file and the logical log index file according to the pipeline sending command when the client type carried in the pipeline sending command indicates that the client is not a data synchronization component, and synchronize the data corresponding to the pipeline sending command to clusters other than the first cluster.

[0237] It should be noted here that the above-mentioned data synchronization device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned data synchronization method embodiment applied to the master node of the first cluster, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0238] The embodiment of the present application further provides a data synchronization device, which is applied to the data synchronization component of the second cluster, such as Fig.11 As shown, the device comprises:

[0239] The first sending module 1101 is used to send a first command to the master node of the first cluster when a connection interruption occurs during the process of synchronizing incremental data between the data synchronization component of the second cluster and the master node of the first cluster, wherein the first command includes a Redis instance identifier and a data identifier, wherein the Redis instance identifier is an identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicates the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment;

[0240] The first receiving module 1102 is used to receive target data synchronized by the master node of the first cluster, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier;

[0241] The second sending module 1103 is used to forward the target data to the master node corresponding to the data synchronization component in the second cluster.

[0242] Optionally, the device further comprises:

[0243] The second receiving module is used to receive the logical log offset corresponding to the target data currently synchronized when receiving the target data.

[0244] Optionally, the device further comprises:

[0245] A third sending module is used to send a second command to the master node of the first cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data;

[0246] The third receiving module is used to receive the Redis instance identifier sent by the master node of the first cluster.

[0247] Optionally, the second sending module 1103 includes:

[0248] A screening submodule, configured to screen the data to be transmitted in the target data based on the data fragmentation identifier corresponding to each piece of data in the target data;

[0249] A determination submodule, used to determine a data channel for transmitting each piece of data to be transmitted according to a database identifier, a data shard identifier and a hash value corresponding to a key value of each piece of data to be transmitted, and a predefined number of data channels N;

[0250] The sending submodule is used to transmit the to-be-transmitted data corresponding to the data channel to the master node of the second cluster in parallel and asynchronously on each of the data channels.

[0251] It should be noted here that the above-mentioned data synchronization device provided in the embodiment of the present application can implement all the method steps implemented in the data synchronization method embodiment of the data synchronization component applied to the second cluster, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0252] An embodiment of the present application also provides a data synchronization device, including a transceiver 1210, a processor 1200, a memory 1220, and a program stored on the memory 1220 and executable on the processor 1200; wherein, when the processor 1200 executes the program, the data synchronization method applied to the master node of the first cluster as described above is implemented, or, the data synchronization method applied to the data synchronization component of the second cluster as described above is implemented.

[0253] The transceiver 1210 is used to receive and send data under the control of the processor 1200.

[0254] Among them, Fig.12 12, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1200 and memory represented by memory 1220. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1210 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium.

[0255] The processor 1200 is responsible for managing the bus architecture and general processing, and the memory 1220 can store data used by the processor 1200 when performing operations.

[0256] The embodiment of the present application also provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the data synchronization method applied to the master node of the first cluster as described above is implemented, or the data synchronization method applied to the data synchronization component of the second cluster as described above is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0257] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for executing the methods described in each embodiment of the present application.

[0258] Therefore, an embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the data synchronization method applied to the master node of the first cluster as described above, or implement the data synchronization method applied to the data synchronization component of the second cluster as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0259] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0260] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data synchronization method, characterized in that: Applied to a master node of a first cluster, the method comprises: Receive a first command sent by a data synchronization component of a second cluster, the first command including a remote dictionary service Redis instance identifier and a data identifier, the data identifier indicating a location of synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment; According to the first command, the target data is obtained; the target data includes: the first incremental data and the second incremental data in the incremental data written by the user, the first incremental data is the data at the location indicated by the data identifier, and the second incremental data is the data after the first incremental data and corresponding to the Redis instance identifier; The target data is synchronized to the data synchronization component of the second cluster in an incremental synchronization manner.

2. The method according to claim 1, characterized in that According to the first command, obtaining target data includes: Search the cache area of ​​the master node to see whether the data corresponding to the data identifier exists. If so, obtain the target data from the cache area. If not, obtain the target data from the distributed storage system corresponding to the master node.

3. The method according to claim 2, characterized in that The method further comprises: When it is detected that incremental data is written into the buffer of the master node, the incremental data written is collected in an asynchronous manner; Generate an identification group corresponding to the incremental data written, wherein the identification group includes at least one of a data shard identification, a data identification, and a database identification; wherein the incremental data written includes one or more incremental data, and each incremental data corresponds to an identification group; The collected incremental data and the identification group corresponding to the incremental data are written into a logical log file, and the logical log index file is updated; wherein the logical log file and the logical log index file are stored in a distributed storage system corresponding to the master node, wherein the logical log index file includes the name of the logical log file and the data identification corresponding to the logical log file.

4. The method according to claim 3, characterized in that: Acquiring the target data in the distributed storage system corresponding to the master node includes: Searching the logical log index file for a logical log file corresponding to the data identifier; In the found logical log file, the target data is retrieved according to the data identifier and the data slice identifier.

5. The method according to claim 3, characterized in that: Writing the incremental data and the identification group corresponding to the incremental data into a logical log file, and updating the logical log index file, including: Loading the logical log index file and the logical log file in the distributed storage system into the memory of the master node; Writing the collected incremental data and the identification group corresponding to the incremental data into the logical log file; Based on the updated logical log file, updating the logical log index file; The updated logical log file and the logical log file index are written into the distributed storage system.

6. The method according to claim 3, characterized in that After writing the collected incremental data and the identification group corresponding to the incremental data into the logical log file and updating the logical log index file, the method further includes: Updating a memory structure corresponding to the logical log file, wherein the memory structure includes a handle corresponding to the logical log file and a file offset currently corresponding to the logical log file; A notification message of a logical log index file update is sent to a standby node corresponding to the master node of the first cluster, wherein the notification message is used to instruct the standby node to update the logical log index file in the standby node memory based on the logical log index file in the distributed storage system, and to update the memory structure in the standby node memory based on the logical log file in the distributed storage system.

7. The method according to claim 6, characterized in that The master node and the standby node corresponding to the master node share a distributed storage system, and the master node has write operation permissions and read operation permissions for the logical log files and logical log index files in the distributed storage system, and the standby node has read operation permissions for the logical log files and logical log index files in the distributed storage system.

8. The method according to claim 1, characterized in that: The method further comprises: When the target data is synchronized to the data synchronization component of the second cluster in an incremental synchronization manner, a logical log offset corresponding to the currently synchronized target data is sent to the data synchronization component of the second cluster.

9. The method according to claim 1, characterized in that: Before receiving the first command sent by the data synchronization component of the second cluster, the method further includes: Receive a second command sent by the data synchronization component of the second cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data; In response to the second command, a Redis instance identifier corresponding to the incremental data currently synchronized between the first cluster and the second cluster is sent to the data synchronization component of the second cluster.

10. The method according to claim 1, characterized in that The method further comprises: Receive the pipeline sending command from the client; When the client type carried in the pipeline sending command indicates that the client is a data synchronization component, executing an operation corresponding to the pipeline sending command; When the client type carried in the pipeline sending command indicates that the client is not a data synchronization component, the logical log file and the logical log index file are updated according to the pipeline sending command, and the data corresponding to the pipeline sending command is synchronized to clusters other than the first cluster.

11. A data synchronization method, characterized in that: The data synchronization component applied to the second cluster, the method comprising: In the event that a connection is interrupted during the process of synchronizing incremental data between the data synchronization component of the second cluster and the master node of the first cluster, a first command is sent to the master node of the first cluster, where the first command includes a Redis instance identifier and a data identifier, where the Redis instance identifier is an identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicates the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment; Receive target data synchronized by the master node of the first cluster, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier; The target data is forwarded to a master node in the second cluster corresponding to the data synchronization component.

12. The method according to claim 11, characterized in that The method further comprises: When receiving the target data, a logical log offset corresponding to the target data currently synchronized is received.

13. The method according to claim 11, characterized in that Before sending the first command to the master node of the first cluster, the method further includes: Sending a second command to the master node of the first cluster, where the second command is used to request a Redis instance identifier corresponding to the currently synchronized incremental data; Receive the Redis instance identifier sent by the master node of the first cluster.

14. The method according to claim 11, characterized in that Forwarding the target data to a master node in the second cluster corresponding to the data synchronization component includes: Filtering the data to be transmitted in the target data based on the data shard identifier corresponding to each piece of data in the target data; Determine a data channel for transmitting each piece of data to be transmitted according to a database identifier, a data shard identifier, a hash value corresponding to a key value of each piece of data to be transmitted, and a predefined number of data channels N; On each of the data channels, the data to be transmitted corresponding to the data channel is asynchronously transmitted in parallel to the master node of the second cluster.

15. A data synchronization device, characterized in that: Applied to a master node of a first cluster, the device comprises: A first receiving module, configured to receive a first command sent by a data synchronization component of a second cluster, wherein the first command includes a remote dictionary service Redis instance identifier and a data identifier, wherein the data identifier indicates a location of synchronized data; wherein the second cluster and the first cluster are deployed in a remote environment; an acquisition module, configured to acquire target data according to the first command; the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data being data at the location indicated by the data identifier, and the second incremental data being data after the first incremental data and corresponding to the Redis instance identifier; The first sending module is used to synchronize the target data to the data synchronization component of the second cluster in an incremental synchronization manner.

16. A data synchronization device, characterized in that: A data synchronization component applied to a second cluster, the device comprising: A first sending module is used to send a first command to the master node of the first cluster when a connection interruption occurs during the process of synchronizing incremental data between the data synchronization component of the second cluster and the master node of the first cluster, wherein the first command includes a Redis instance identifier and a data identifier, wherein the Redis instance identifier is an identifier of the Redis instance corresponding to the incremental data currently synchronized between the first cluster and the second cluster, and the data identifier indicates the location of the synchronized data; the second cluster and the first cluster are deployed in a remote environment; A first receiving module is used to receive target data synchronized by the master node of the first cluster, wherein the target data includes: first incremental data and second incremental data in the incremental data written by the user, the first incremental data is data at the location indicated by the data identifier, and the second incremental data is data after the first incremental data and corresponding to the Redis instance identifier; The second sending module is used to forward the target data to the master node corresponding to the data synchronization component in the second cluster.

17. A data synchronization device, comprising a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the processor implements the data synchronization method according to any one of claims 1 to 10, or implements the data synchronization method according to any one of claims 11 to 14.

18. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the data synchronization method according to any one of claims 1 to 10 is implemented, or the data synchronization method according to any one of claims 11 to 14 is implemented.

19. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the data synchronization method as claimed in any one of claims 1 to 10, or implement the data synchronization method as claimed in any one of claims 11 to 14.

Images