Redis Cluster Inter-Cluster Data Synchronization Method, Apparatus and System

Abstract

The present application provides a method, apparatus, and system for data synchronization between Redis clusters, which relates to the technical field of data transmission and can be used in the financial field. The method includes: obtaining source cluster shard information to be synchronized, and starting a corresponding number of receiving threads to receive user data; obtaining target cluster shard information to be synchronized, starting a corresponding number of sending threads to send user data, and creating a cache for each target cluster shard according to the target cluster shard information to save the command data processed by the source cluster shards; forwarding the database snapshots in the received source cluster shards to the corresponding target cluster shards; merging the command data in the cache, and sending the merged result to the corresponding target cluster shards through the sending threads. The present application realizes merged transmission of the command data in the cache, which can save bandwidth and reduce the decline in the cluster service capacity caused by frequent triggering of full synchronization due to network jitter.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and can be used in the financial field, and particularly relates to a method, device, and system for data synchronization between Redis clusters. Background Art

[0002] In business scenarios such as finance that require a high level of high availability, to prevent and resist the devastating blows and incalculable losses brought by unexpected situations, it is guaranteed for reliable operation of important business systems to have disaster backup.

[0003] The Redis database is a memory database widely used in the industry. At present, some users use the open-source product redis-shake to implement data synchronization between Redis clusters. However, the positioning of redis-shake is a small tool, which does not have high availability itself, cannot automatically recover after abnormal interruption of data transmission, cannot self-adapt after changes in source and destination cluster shards, and does not have the ability to integrate and compress transmitted data, which will occupy a large bandwidth. When network jitter occurs, etc., it is easy to trigger full synchronization, which will bring a large load pressure to the source cluster. Summary of the Invention

[0004] In view of this, the present invention provides a method, device, and system for data synchronization between Redis clusters to solve at least one of the above-mentioned problems.

[0005] To achieve the above object, the present invention adopts the following solutions:

[0006] According to a first aspect of the present invention, there is provided a method for data synchronization between Redis clusters, the method including: obtaining Redis source cluster shard information to be synchronized, starting a corresponding number of receiving threads according to the source cluster shard information to receive user data, each receiving thread corresponding to a shard in the source cluster; obtaining Redis target cluster shard information to be synchronized, starting a corresponding number of sending threads according to the target cluster shard information to send user data, and creating a cache for each target cluster shard according to the target cluster shard information to save command data processed by the Redis source cluster shards; forwarding database snapshots in the Redis source cluster shards received by the receiving threads to corresponding Redis target cluster shards, so that the Redis target cluster shards can load data in the database snapshots into local databases; merging the command data in the cache according to a preset rule, and sending the merging result to the corresponding Redis target cluster shards through the sending threads.

[0007] According to a second aspect of the present invention, there is provided a data synchronization device between Redis clusters, the device comprising: a receiving thread creation unit, configured to obtain the sharding information of the Redis source cluster to be synchronized, and start a corresponding number of receiving threads according to the source cluster sharding information to receive user data, each receiving thread corresponding to a shard in the source cluster; a sending thread creation unit, configured to obtain the sharding information of the Redis target cluster to be synchronized, and start a corresponding number of sending threads according to the target cluster sharding information to send user data; a cache creation unit, configured to create a cache for each target cluster shard according to the target cluster sharding information to store the command data processed by the Redis source cluster shards; a full amount information sending unit, configured to forward the database snapshot in the Redis source cluster shards received by the receiving threads to the corresponding Redis target cluster shards, so that the Redis target cluster shards can load the data in the database snapshot into the local database; a command merging and sending unit, configured to merge the command data in the cache according to a preset rule, and send the merging result to the corresponding Redis target cluster shards through the sending threads.

[0008] According to a third aspect of the present invention, there is provided a data synchronization system between Redis clusters, the system comprising: a monitoring system, a registration center, a synchronization component, a Redis source cluster, and a Redis target cluster, wherein the monitoring system is communicatively connected to the registration center, the synchronization component, the Redis source cluster, and the Redis target cluster respectively, the synchronization component is also communicatively connected to the Redis source cluster and the Redis target cluster respectively, the registration center stores the synchronization component information, the sharding information of the Redis source cluster, and the sharding information of the Redis target cluster, the monitoring system monitors the running states of the synchronization component, the Redis source cluster, and the Redis target cluster, and monitors the information change of the registration center, the synchronization component is used for performing data synchronization between the Redis source cluster and the Redis target cluster, and the synchronization component implements the steps of the data synchronization method between Redis clusters as described above.

[0009] According to a fourth aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data synchronization method between Redis clusters as described above are implemented.

[0010] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data synchronization method between Redis clusters as described above are implemented.

[0011] According to a sixth aspect of the present invention, there is provided a computer program product including computer programs / instructions, which when executed by a processor implement the steps of the Redis cluster - to - cluster data synchronization method as described above.

[0012] As described above, since the present application sets a cache for command data during Redis cluster - to - cluster data synchronization and implements combined transmission for the command data in the cache, it can save bandwidth and greatly reduce the decline in cluster service capabilities caused by frequent triggering of full - volume synchronization due to network jitter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0014] Figure 1 is a schematic structural diagram of a Redis cluster - to - cluster data synchronization system provided by an embodiment of the present application;

[0015] Figure 2 is a schematic flowchart of a Redis cluster - to - cluster data synchronization method provided by an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of cluster expansion provided by an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of cluster contraction provided by an embodiment of the present application;

[0018] Figure 5 is a schematic diagram of breakpoint - resuming transmission provided by an embodiment of the present application;

[0019] Figure 6 is a schematic diagram of failover provided by an embodiment of the present application;

[0020] Figure 7 is a schematic structural diagram of a Redis cluster - to - cluster data synchronization device provided by an embodiment of the present application;

[0021] Figure 8 is a schematic block diagram of the system composition of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the following further elaborates on the embodiments of the present invention with reference to the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0023] It should be noted that a method, device, and system for data synchronization between Redis clusters disclosed in this application can be used in the financial field and can also be used in any field other than the financial field. This application does not limit the application fields of the disclosed method, device, and system for data synchronization between Redis clusters.

[0024] As Figure 1 shown is a schematic structural diagram of a system for data synchronization between Redis clusters provided by an embodiment of this application. The system includes: a monitoring system 100, a registration center 200, a Redis source cluster 300, a Redis target cluster 400, and a synchronization component 500. The monitoring system 100 is respectively communicatively connected to the registration center 200, the synchronization component 500, the Redis source cluster 300, and the Redis target cluster 400. The synchronization component 500 is also respectively communicatively connected to the Redis source cluster 300 and the Redis target cluster 400.

[0025] In this embodiment, the registration center 200 is responsible for storing information, and it can store the following information: (1) Redis source cluster information, such as source cluster node IP, port, sharding, sharding running status, etc.; (2) Redis target cluster information, such as target cluster node IP, port, sharding, sharding running status, etc.; (3) synchronization component node IP, port, sharding information of the responsible source cluster and target cluster, synchronization component running status, etc.

[0026] The monitoring system 100 monitors the running status of the synchronization component 100, the Redis source cluster 300, and the Redis target cluster 400, and simultaneously monitors information changes in the registration center 200, and it can perform read and write operations on the registration center 200.

[0027] The synchronization component 500 is used to complete data synchronization between the source cluster sharding and the target cluster sharding it is responsible for. As Figure 2 shown, the synchronization component 500 completes the above data synchronization through the following steps:

[0028] Step S201: Obtain the Redis source cluster sharding information to be synchronized, and start a corresponding number of receiving threads to receive user data according to the source cluster sharding information. Each receiving thread corresponds to a sharding in the source cluster.

[0029] In this embodiment, the sharding information of the Redis source cluster to be synchronized can be pre-stored in the configuration file of the synchronization component 500, or can be sent by an external system, such as the monitoring system 100, to the synchronization component 500. It may include, but is not limited to: source cluster sharding IP addresses, port information, sharding information, sharding running status and other information.

[0030] According to the number of shards in the source cluster to be synchronized, start the same number of receiving threads, and each receiving thread corresponds to a shard.

[0031] Step S202: Obtain the sharding information of the Redis target cluster to be synchronized, start the corresponding number of sending threads according to the target cluster sharding information to send user data, and create a cache for each target cluster shard according to the target cluster sharding information to save the command data processed by the Redis source cluster shards.

[0032] In this embodiment, the sharding information of the Redis target cluster to be synchronized is the same as the above-mentioned Redis source cluster sharding information. It can be pre-stored in the configuration file of the synchronization component 500, or can be sent by an external system, such as the monitoring system 100, to the synchronization component. It may include, but is not limited to: target cluster sharding IP addresses, port information, sharding information, sharding running status and other information.

[0033] In this embodiment, one shard in the source cluster corresponds to one shard in the target cluster. Therefore, the number of sending threads and receiving threads started in this step is also the same.

[0034] Step S203: Forward the database snapshot in the Redis source cluster shards received by the receiving threads to the corresponding Redis target cluster shards, so that the Redis target cluster shards can load the data in the database snapshot into the local database;

[0035] When performing full synchronization, each shard in the Redis source cluster will send the current database snapshot to the synchronization component. After the synchronization component receives the database snapshot through the receiving threads, it directly forwards it to the corresponding shards in the Redis target cluster through the sending threads. After the Redis target cluster shards receive the database snapshot, they load the data into their local databases, thus completing the full data synchronization.

[0036] Step S204: Merge the command data in the cache according to the preset rules, and send the merged result to the corresponding Redis target cluster shards through the sending threads.

[0037] Preferably, each data operation type in the command data can correspond to a map, with the key of the operation as the key of the corresponding map, and the data operations for the same key are merged. Here, the map is a common element storage method provided in Java. It is a collection class. The map collection class is used to store key-value pairs, that is, Map<key, value>, and each key is mapped to a value. In this step, sending the merged result to the corresponding Redis target cluster shard through the sending thread can include: sending the merged result to the corresponding Redis target cluster shard according to a preset time interval, and this preset time interval can be set according to the actual situation, such as 10ms, etc.

[0038] In this embodiment, for example, merge operations can be performed on commands of string register type requests, string counter type, set type, linked list type, hash table type, sorted set, etc.

[0039] It should be noted that this embodiment does not limit the order between the above step S201 and step S202, that is, they can be carried out sequentially or simultaneously. And this embodiment also does not limit the order between step S203 and step S204, and step 3 and step 4 can be carried out simultaneously. Figure 2 The shown order is only the execution order of one embodiment of this application.

[0040] As described above, the Redis cluster inter-data synchronization system provided by this application, because a cache is set for the command data during the Redis cluster inter-data synchronization and the merged transmission of the command data in the cache is realized, it can achieve the saving of bandwidth and greatly reduce the decline of the cluster service ability caused by the frequent triggering of full synchronization due to network jitter.

[0041] Preferably, in this embodiment, when the situation of cluster expansion occurs, such as Figure 3 As shown, the monitoring system 100 perceives the change of the storage shard by listening to the registration center 200, obtains the information of the newly added source cluster shard and the target cluster shard from the registration center 200, and provides the newly added shard information to the synchronization component 500. At the same time, the monitoring system will perform a load balancing operation on the shards in the source cluster and transfer the relevant services to the new shards. After receiving the newly added shard information, the synchronization component 500 will create a receiving process and a sending process according to the newly added shard information, and synchronize the data in the newly added source cluster shard after load balancing to the newly added target cluster shard. When synchronizing, the operations of the above steps S203 and S204 can be adopted.

[0042] Preferably, in this embodiment, when the situation of cluster contraction occurs, such as Figure 4As shown, by listening to the registry 200, the monitoring system 100 senses a change in the storage shard, migrates the data in the shards to be reduced in the source cluster to other shards through load balancing, and notifies the synchronization component 500 to synchronize the changes in the source cluster data to the target cluster. After the data migration and data synchronization are completed, the monitoring system 100 shuts down the shards to be scaled down in the source cluster and the shards to be scaled down in the target cluster, and notifies the synchronization component of the shard information that has been removed. The synchronization component 500 deletes the corresponding receiving and sending threads according to the removed shard information to release resources.

[0043] It can be seen that by establishing a listening mechanism between the synchronization component and the registry through the monitoring system, the self-adaptation of multiple shard nodes in the source cluster and the target cluster and the ability to trigger data synchronization can be completed.

[0044] Preferably, in this embodiment, when a certain shard in the target cluster fails, such as Figure 5 As shown, after the failed shard ( Figure 5 shard 1 in ) in the target cluster restarts, it sends an incremental synchronization command (psync) to the synchronization component 500. The incremental synchronization command contains source shard identification information and synchronized offset position information. The synchronization component 500 is provided with a cache. This cache and the cache in step S204 can be the same or set separately. This cache can be an aof-binlog cache, and the source shard identification information, starting offset position information, and sending position information are stored in the cache. After receiving the incremental synchronization command, if the synchronized offset position information is greater than the starting offset position information of the cache, the synchronization component 500 sets the sending position in the cache to the synchronized offset position of the failed shard in the target cluster and starts the sending thread to send the missing data of the failed shard; if the synchronized offset position information is less than or equal to the starting offset position information of the cache, a full synchronization of the failed shard is triggered, and the shard data in the source cluster is fully synchronized to the corresponding failed shard.

[0045] Through the above cache setting, the synchronization component 500 can have the ability to resume interrupted transfer, avoid the performance loss caused by excessive full synchronization to the source cluster, and improve the continuity of the service.

[0046] Preferably, in this embodiment, when a certain synchronization component fails, such as Figure 6As shown in the figure, when the monitoring system 100 detects a failure of the synchronization component 501, it finds a normally operating synchronization component 502 through the registration center 200 and sends a takeover node command to it. The takeover node command contains the shard information of the source cluster to be taken over and the shard information of the target cluster to be taken over. After receiving the takeover node command, the synchronization component 502 creates a receiving thread and a sending thread, synchronizes the data in the shards of the source cluster to be taken over to the shards of the target cluster to be taken over, and at the same time feeds back the synchronization result to the monitoring system 100, and the monitoring system 100 updates the relevant information in the registration center 200.

[0047] In this embodiment, through the collaborative scheduling of the monitoring system and the registration center, the synchronization component can achieve high-availability capabilities such as automatic takeover and self-healing when some nodes of the synchronization component go down.

[0048] An embodiment of the present application further provides a method for synchronizing data between Redis clusters. This embodiment elaborates on this method from the perspective of the synchronization component. The method includes:

[0049] Step 1: Obtain the shard information of the Redis source cluster to be synchronized, and start a corresponding number of receiving threads according to the shard information of the source cluster to receive user data. Each receiving thread corresponds to a shard in the source cluster;

[0050] Step 2: Obtain the shard information of the Redis target cluster to be synchronized, start a corresponding number of sending threads according to the shard information of the target cluster to send user data, and create a cache for each shard of the target cluster according to the shard information of the target cluster to save the command data processed by the shards of the Redis source cluster;

[0051] Step 3: Forward the database snapshot in the shards of the Redis source cluster received by the receiving thread to the corresponding shards of the Redis target cluster, so that the shards of the Redis target cluster can load the data in the database snapshot into the local database;

[0052] Step 4: Merge the command data in the cache according to a preset rule, and send the merged result to the corresponding shards of the Redis target cluster through the sending thread.

[0053] It should be noted that this embodiment does not limit the order between the above steps 1 and 2, that is, they can be carried out sequentially or simultaneously, and this embodiment also does not limit the order between steps 3 and 4. Steps 3 and 4 can be carried out simultaneously.

[0054] Preferably, in step 4 above, merging the command data in the cache according to a preset rule may include: corresponding a map to each data operation type in the command data, using the key of the operation as the key of the corresponding map, and merging the data operations for the same key; sending the merged result to the corresponding Redis target cluster shard by the sending thread includes: sending the merged result to the corresponding Redis target cluster shard at preset time intervals.

[0055] Preferably, the Redis cluster - to - cluster data synchronization method provided in this embodiment may further include: receiving the new shard information sent by the monitoring system, where the new shard information includes the new source cluster shard information and the new target cluster shard information; creating a receiving thread and a sending thread according to the new shard information, and synchronizing the data in the load - balanced new source cluster shard to the new target cluster shard.

[0056] Preferably, the Redis cluster - to - cluster data synchronization method provided in this embodiment may further include: synchronizing the shard data in the source cluster that has been load - balanced due to shard removal to the corresponding shard in the target cluster; receiving the shard removal information sent by the monitoring system, where the shard removal information includes the removed source cluster shard information and the removed target cluster shard information; deleting the corresponding receiving thread and sending thread according to the shard removal information to release resources.

[0057] Preferably, the Redis cluster - to - cluster data synchronization method provided in this embodiment may further include: receiving the incremental synchronization command sent after the restart of the failed shard in the target cluster, where the incremental synchronization command includes the source node identification information and the synchronized offset position information; finding the corresponding buffer according to the source node identification information, and reading the starting offset position information in the cache; in response to the synchronized offset position information being greater than the starting offset position information in the cache, setting the sending position in the cache to the synchronized offset position of the failed shard in the target cluster, and starting the sending thread to send the missing data of the failed shard; in response to the synchronized offset position information being less than or equal to the starting offset position information in the cache, triggering a full - volume synchronization corresponding to the failed shard.

[0058] Preferably, the cache in the Redis cluster - to - cluster data synchronization method provided in this embodiment is an aof - binlog cache.

[0059] Preferably, the Redis cluster - to - cluster data synchronization method provided in this embodiment may further include: receiving the take - over node command issued by the monitoring system, where the take - over node command contains the source cluster shard information to be taken over and the target cluster shard information to be taken over; creating a receiving thread and a sending thread according to the take - over node command, and synchronizing the data in the source cluster shard to be taken over to the target cluster shard to be taken over.

[0060] As described above, in the Redis cluster - to - cluster data synchronization method provided by this application, since command data is cached during Redis cluster - to - cluster data synchronization and the command data in the cache is merged and transmitted, it is possible to save bandwidth and greatly reduce the decline in cluster service capabilities caused by frequent triggering of full - volume synchronization due to network jitter. In addition, through the collaborative scheduling of the monitoring component and the registration center, this application can achieve high - availability capabilities such as automatic takeover and self - healing when some nodes of the synchronization component are down, and can also support the dynamic elastic scaling of Redis clusters. In addition, when the target shard fails and restarts, it can also have the ability to resume from the breakpoint, avoiding the performance loss caused by full - volume synchronization to the main cluster and improving service continuity.

[0061] As Figure 7 As shown in the figure, another embodiment of this application also provides a Redis cluster - to - cluster data synchronization device, which includes: an information acquisition unit 710, a thread creation unit 720, a cache creation unit 730, a full - volume information sending unit 740, and a command merging and sending unit 750. Among them, the thread creation unit 720 is respectively connected to the information acquisition unit 710, the full - volume information sending unit 740, and the command merging and sending unit 750, and the cache creation unit 730 is respectively connected to the information acquisition unit 710 and the command merging and sending unit 750.

[0062] The information acquisition unit 710 is used to acquire the shard information of the Redis source cluster and the shard information of the Redis target cluster that need to be synchronized.

[0063] The thread creation unit 720 is used to start a corresponding number of receiving threads to receive user data according to the source cluster shard information, with each receiving thread corresponding to a shard in the source cluster, and start a corresponding number of sending threads to send user data according to the target cluster shard information.

[0064] The cache creation unit 730 is used to create a cache for each target cluster shard according to the target cluster shard information to save the command data processed by the Redis source cluster shard.

[0065] The full - volume information sending unit 740 is used to forward the database snapshot in the Redis source cluster shard received by the receiving thread to the corresponding Redis target cluster shard, so that the Redis target cluster shard can load the data in the database snapshot into the local database.

[0066] The command merging and sending unit 750 is used to merge the command data in the cache according to a preset rule and send the merging result to the corresponding Redis target cluster shard through the sending thread.

[0067] Preferably, the command merging and sending unit 750 is specifically configured to: correspond a map to each data operation type in the command data, use the key of the operation as the key of the corresponding map, and merge the data operations on the same key; and send the merging result to the corresponding Redis target cluster shard according to a preset time interval.

[0068] Preferably, the Redis cluster - to - cluster data synchronization device in this embodiment further includes: an expansion information receiving unit, configured to receive the new shard information sent by the monitoring system, where the new shard information includes new source cluster shard information and new target cluster shard information; a thread creation unit 720 configured to create a receiving thread and a sending thread according to the new shard information, and the full - amount information sending unit 740 and the command merging and sending unit 750 are configured to synchronize the data in the load - balanced new source cluster shards to the new target cluster shards.

[0069] Preferably, the Redis cluster - to - cluster data synchronization device in this embodiment further includes: a shrinkage information receiving unit, configured to receive the removed shard information sent by the monitoring system, where the removed shard information includes removed source cluster shard information and removed target cluster shard information; the full - amount information sending unit 740 and the command merging and sending unit 750 are configured to synchronize the shard data in the source cluster that has been load - balanced due to the removed shard to the shards in the corresponding target cluster; a thread release unit, configured to delete the corresponding receiving thread and sending thread according to the removed shard information to release resources.

[0070] Preferably, the Redis cluster - to - cluster data synchronization device in this embodiment further includes: a synchronization command receiving unit, configured to receive the incremental synchronization command sent after the restart of the failed shard in the target cluster, where the incremental synchronization command includes source node identification information and the synchronized offset position information; a position reading unit, configured to find the corresponding buffer according to the source node identification information and read the starting offset position information in the buffer; a breakpoint - resumed transmission unit, configured to, in response to the synchronized offset position information being greater than the starting offset position information in the buffer, set the sending position in the buffer to the synchronized offset position of the failed shard in the target cluster and start a sending thread to send the missing data of the failed shard; a full - amount synchronization unit, configured to, in response to the synchronized offset position information being less than or equal to the starting offset position information in the buffer, trigger the full - amount synchronization corresponding to the failed shard.

[0071] Preferably, the buffer in the Redis cluster - to - cluster data synchronization device in this embodiment is an aof - binlog buffer.

[0072] Preferably, the Redis cluster - to - cluster data synchronization device in this embodiment further includes: a takeover command receiving unit, configured to receive a takeover node command issued by a monitoring system, where the takeover node command includes source cluster shard information to be taken over and target cluster shard information to be taken over; a full - volume information sending unit 740 and a command merging and sending unit 750 are configured to create a receiving thread and a sending thread according to the takeover node command, and synchronize the data in the source cluster shard to be taken over to the target cluster shard to be taken over.

[0073] As described above, for the Redis cluster - to - cluster data synchronization device provided in this application, since a cache is set for command data during Redis cluster - to - cluster data synchronization and the command data in the cache is merged and transmitted, bandwidth can be saved, and the decline in cluster service capabilities caused by frequent triggering of full - volume synchronization due to network jitter can be greatly reduced. In addition, through the collaborative scheduling of the monitoring component and the registration center in this application, high - availability capabilities such as automatic takeover and self - healing can be achieved when some nodes of the synchronization component are down, and dynamic elastic scaling of the Redis cluster can also be supported. In addition, when the target shard fails and restarts, the ability to resume from breakpoint can also be available, avoiding the performance loss caused by full - volume synchronization to the main cluster and improving service continuity.

[0074] Figure 8 It is a schematic block diagram of the system composition of an electronic device 1100 provided in another embodiment of the present invention. As Figure 8 shown, the electronic device 1100 may include a central processing unit 1110 and a memory 1120; the memory 1120 is coupled to the central processing unit 1110. It should be noted that this Figure 8 is exemplary; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0075] In one embodiment, the function of the Redis cluster - to - cluster data synchronization method can be integrated into the central processing unit 1110. Among them, the central processing unit 1110 can be configured to perform the following controls:

[0076] Step S201: Obtain the Redis source cluster shard information to be synchronized, and start corresponding numbers of receiving threads according to the source cluster shard information to receive user data, where each receiving thread corresponds to a shard in the source cluster;

[0077] Step S202: Obtain the Redis target cluster shard information to be synchronized, start corresponding numbers of sending threads according to the target cluster shard information to send user data, and create a cache for each target cluster shard according to the target cluster shard information to save the command data processed by the Redis source cluster shard;

[0078] Step S203: Forward the database snapshot in the Redis source cluster shards received by the receiving thread to the corresponding Redis target cluster shards, so that the Redis target cluster shards can load the data in the database snapshot into the local database;

[0079] Step S204: Merge the command data in the cache according to a preset rule, and send the merge result to the corresponding Redis target cluster shards through the sending thread.

[0080] In another embodiment, the Redis cluster - to - cluster data synchronization device can be separately configured from the central processing unit 1110. For example, the Redis cluster - to - cluster data synchronization device can be configured as a chip connected to the central processing unit 1110, and the above - mentioned Redis cluster - to - cluster data synchronization method functions are realized through the control of the central processing unit.

[0081] As Figure 8 shown, the electronic device 1100 may further include: a communication module 1130, an input unit 1140, an audio processor 1150, a display 1160, and a power supply 1170. It should be noted that the electronic device 1100 does not necessarily have to include all the components shown in Figure 8 ; in addition, the electronic device 1100 may further include components not shown in Figure 8 , and reference can be made to the prior art.

[0082] As Figure 8 shown, the central processing unit 1110, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor devices and / or logic devices. The central processing unit 1110 receives inputs and controls the operations of the various components of the electronic device 1100.

[0083] Among them, the memory 1120 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non - volatile memory, or other suitable devices. It can store information related to the above - mentioned transfer method, and can also store programs for executing the relevant information. And the central processing unit 1110 can execute the program stored in the memory 1120 to implement information storage or processing, etc.

[0084] The input unit 1140 provides inputs to the central processing unit 1110. The input unit 1140 is, for example, a key or a touch input device. The power supply 1170 is used to supply power to the electronic device 1100. The display 1160 is used for displaying display objects such as images and texts. The display can be, for example, an LCD display, but is not limited thereto.

[0085] The memory 1120 can be a solid-state memory, for example, a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that stores information even when power is off, can be selectively erased and has more data. Examples of such a memory are sometimes referred to as EPROM, etc. The memory 1120 can also be some other type of device. The memory 1120 includes a buffer memory 1121 (sometimes referred to as a buffer). The memory 1120 can include an application / function storage unit 1122, which is used to store application programs and function programs or the processes for operating the electronic device 1100 by the central processor 1110.

[0086] The memory 1120 can also include a data storage unit 1123, which is used to store data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 1124 of the memory 1120 can include various drivers of the electronic device for communication functions and / or for performing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0087] The communication module 1130 is a transmitter / receiver 1130 that transmits and receives signals via the antenna 1131. The communication module (transmitter / receiver) 1130 is coupled to the central processor 1110 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0088] Based on different communication technologies, multiple communication modules 1130 can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module (transmitter / receiver) 1130 is also coupled to the speaker 1151 and the microphone 1152 via the audio processor 1150 to provide an audio output via the speaker 1151 and receive an audio input from the microphone 1152, so as to achieve normal telecommunication functions. The audio processor 1150 can include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 1150 is also coupled to the central processor 1110, so that it is possible to record on the device via the microphone 1152 and play the sound stored on the device via the speaker 1151.

[0089] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps in the Redis cluster inter-data synchronization method where the execution entity in the above embodiments is a client or a server. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, all steps of the Redis cluster inter-data synchronization method where the execution entity is a client or a server in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0090] Step S201: Obtain the shard information of the Redis source cluster to be synchronized, and start a corresponding number of receiving threads according to the source cluster shard information to receive user data. Each receiving thread corresponds to a shard in the source cluster;

[0091] Step S202: Obtain the shard information of the Redis target cluster to be synchronized, start a corresponding number of sending threads according to the target cluster shard information to send user data, and create a cache for each target cluster shard according to the target cluster shard information to save the command data processed by the Redis source cluster shard;

[0092] Step S203: Forward the database snapshot in the Redis source cluster shard received by the receiving thread to the corresponding Redis target cluster shard, so that the Redis target cluster shard can load the data in the database snapshot into the local database;

[0093] Step S204: Merge the command data in the cache according to a preset rule, and send the merge result to the corresponding Redis target cluster shard through the sending thread.

[0094] As can be seen from the above description, for the computer-readable storage medium provided by the embodiments of the present application, since a cache is set for the command data during the Redis cluster inter-data synchronization and the merged transmission of the command data in the cache is realized, bandwidth can be saved, and the decline of the cluster service ability caused by frequent triggering of full synchronization due to network jitter can be greatly reduced. In addition, through the collaborative scheduling of the monitoring component and the registration center in the present application, high-availability capabilities such as automatic takeover and self-healing can be realized when some nodes of the synchronization component go down, and dynamic elastic scaling of the Redis cluster can also be supported. In addition, when the target shard fails and restarts, it can also have the ability to resume from a breakpoint, avoiding the performance loss caused by full synchronization to the main cluster and improving service continuity.

[0095] An embodiment of the present application further provides a computer program product that can implement all steps of the Redis cluster inter - data synchronization method in which the execution subject in the above - mentioned embodiment is a client or a server. When the computer program / instructions are executed by a processor, the steps of the Redis cluster inter - data synchronization method are implemented. For example, the computer program / instructions implement the following steps:

[0096] Step S201: Obtain the shard information of the Redis source cluster to be synchronized, and start a corresponding number of receiving threads according to the source cluster shard information to receive user data. Each receiving thread corresponds to a shard in the source cluster;

[0097] Step S202: Obtain the shard information of the Redis target cluster to be synchronized, start a corresponding number of sending threads according to the target cluster shard information to send user data, and create a cache for each target cluster shard according to the target cluster shard information to save the command data processed by the Redis source cluster shards;

[0098] Step S203: Forward the database snapshot in the Redis source cluster shards received by the receiving threads to the corresponding Redis target cluster shards, so that the Redis target cluster shards can load the data in the database snapshot into the local database;

[0099] Step S204: Merge the command data in the cache according to a preset rule, and send the merged result to the corresponding Redis target cluster shards through the sending threads.

[0100] As can be seen from the above description, for the computer program product provided by the embodiment of the present application, since a cache is set for the command data during the Redis cluster inter - data synchronization and the merged transmission of the command data in the cache is implemented, bandwidth can be saved, and the decline of the cluster service ability caused by the frequent triggering of full - volume synchronization due to network jitter can be greatly reduced. In addition, through the collaborative scheduling of the monitoring component and the registration center in the present application, high - availability capabilities such as automatic takeover and self - healing can be achieved when some nodes of the synchronization component go down, and the dynamic elastic scaling of the Redis cluster can also be supported. In addition, when the target shard fails and restarts, it can also have the ability to resume data transfer from the breakpoint, avoiding the performance loss caused by full - volume synchronization to the main cluster and improving service continuity.

[0101] In the description of this specification, the descriptions referring to terms such as "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present invention, and the order of steps is not limited and can be adjusted appropriately as needed.

[0102] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0103] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0104] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby the instructions executed on the computer or other programmable apparatus provide steps for realizing the functions specified in one process or a plurality of processes and / or one block or a plurality of blocks in the flow Figure 1 one process or a plurality of processes and / or Figure 1 steps for realizing the functions specified in one block or a plurality of blocks.

[0106] In the above specific embodiments, the objectives, technical solutions and beneficial effects of the present invention have been further described in detail. It should be understood that the above are only specific embodiments of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data synchronization between Redis clusters, characterized in that, The method includes: Obtain the sharding information of the Redis source cluster to be synchronized, and start a corresponding number of receiving threads according to the source cluster sharding information to receive user data, where each receiving thread corresponds to a shard in the source cluster; Obtain the sharding information of the Redis target cluster to be synchronized, start a corresponding number of sending threads according to the target cluster sharding information to send user data, and create a cache for each target cluster shard according to the target cluster sharding information to save the command data processed by the Redis source cluster shard; Forward the database snapshot in the Redis source cluster shard received by the receiving thread to the corresponding Redis target cluster shard, so that the Redis target cluster shard loads the data in the database snapshot into the local database; Merge the command data in the cache according to a preset rule, and send the merged result to the corresponding Redis target cluster shard through the sending thread; Receive the incremental synchronization command sent after the failed shard of the target cluster restarts, where the incremental synchronization command includes source node identification information and synchronized offset position information; Find the corresponding buffer according to the source node identification information, and read the starting offset position information in the cache. The source shard identification information and sending position information are also stored in the cache; In response to the synchronized offset position information being greater than the starting offset position information in the cache, set the sending position in the cache to the synchronized offset position of the failed shard in the target cluster, and start the sending thread to send the data missing from the failed shard; In response to the synchronized offset position information being less than or equal to the starting offset position information in the cache, trigger a full synchronization corresponding to the failed shard, and fully synchronize the shard data in the Redis source cluster to the corresponding failed shard.

2. The Redis cluster - to - cluster data synchronization method according to claim 1, characterized in that, The merging the command data in the cache according to a preset rule includes: corresponding each data operation type in the command data to a map, using the key of the operation as the key of the corresponding map, and merging the data operations on the same key; the sending the merged result to the corresponding Redis target cluster shard through the sending thread includes: sending the merged result to the corresponding Redis target cluster shard according to a preset time interval.

3. The Redis cluster - to - cluster data synchronization method according to claim 1, characterized in that, It also includes: Receive the new sharding information sent by the monitoring system, where the new sharding information includes new source cluster sharding information and new target cluster sharding information; Create receiving threads and sending threads according to the new sharding information, and synchronize the data in the load-balanced new source cluster shards to the new target cluster shards.

4. The Redis cluster - to - cluster data synchronization method according to claim 1, wherein It also includes: Synchronize the shard data in the source cluster that has been load-balanced due to the removed shard to the shard in the corresponding target cluster; Receive the removed sharding information sent by the monitoring system, where the removed sharding information includes removed source cluster sharding information and removed target cluster sharding information; Delete the corresponding receiving threads and sending threads according to the removed sharding information to release resources.

5. The Redis cluster - to - cluster data synchronization method according to claim 1, characterized in that, The cache is an aof-binlog cache.

6. The Redis cluster - to - cluster data synchronization method according to claim 1, characterized in that, It also includes: Receive the takeover node command sent by the monitoring system, where the takeover node command includes the shard information of the source cluster to be taken over and the shard information of the target cluster to be taken over; Create a receiving thread and a sending thread according to the takeover node command, and synchronize the data in the shards of the source cluster to be taken over to the shards of the target cluster to be taken over.

7. A data synchronization device between Redis clusters, characterized in that, The device includes: An information acquisition unit for acquiring the Redis source cluster shard information and Redis target cluster shard information that need to be synchronized; A thread creation unit for starting a corresponding number of receiving threads according to the source cluster shard information to receive user data, each receiving thread corresponding to a shard in the source cluster, and starting a corresponding number of sending threads according to the target cluster shard information to send user data; A cache creation unit for creating a cache for each target cluster shard according to the target cluster shard information to save the command data processed by the Redis source cluster shard; A full amount information sending unit for forwarding the database snapshot in the Redis source cluster shard received by the receiving thread to the corresponding Redis target cluster shard, so that the Redis target cluster shard loads the data in the database snapshot into the local database; A command merging and sending unit for merging the command data in the cache according to a preset rule, and sending the merging result to the corresponding Redis target cluster shard through the sending thread; A synchronization command receiving unit for receiving the incremental synchronization command sent after the restart of the failed shard of the target cluster, where the incremental synchronization command includes the source node identification information and the synchronized offset position information; Find the corresponding buffer according to the source node identification information, and read the starting offset position information in the cache. The source shard identification information and the sending position information are also stored in the cache; In response to the synchronized offset position information being greater than the starting offset position information in the cache, set the sending position in the cache to the synchronized offset position of the failed shard in the target cluster, and start the sending thread to send the missing data of the failed shard; In response to the synchronized offset position information being less than or equal to the starting offset position information in the cache, trigger a full amount synchronization corresponding to the failed shard, and fully synchronize the shard data in the Redis source cluster to the corresponding failed shard.

8. A data synchronization system between Redis clusters, characterized in that, The system includes: a monitoring system, a registration center, a synchronization component, a Redis source cluster, and a Redis target cluster. Among them, the monitoring system is communicatively connected to the registration center, the synchronization component, the Redis source cluster, and the Redis target cluster respectively. The synchronization component is also communicatively connected to the Redis source cluster and the Redis target cluster respectively. The registration center stores the information of the synchronization component, the sharding information of the Redis source cluster, and the sharding information of the Redis target cluster. The monitoring system monitors the running states of the synchronization component, the Redis source cluster, and the Redis target cluster, and monitors the information change of the registration center. The synchronization component is used to perform data synchronization between the Redis source cluster and the Redis target cluster, and the synchronization component implements the steps of the Redis cluster inter - data synchronization method according to any one of claims 1 - 6.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the Redis cluster inter - data synchronization method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the Redis cluster inter - data synchronization method according to any one of claims 1 to 6.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, it implements the steps of the Redis cluster inter - data synchronization method according to any one of claims 1 to 6.

Images