A fast expansion method of a Redis cluster based on a Proxy architecture

Abstract

This invention discloses a rapid scaling method for a Redis cluster based on a proxy architecture, including: precondition determination; execution plan formulation; backup execution; RDB file transfer; RDB file detection; full data loading; source node write restriction: the source node initiates a write restriction command to ensure no further data addition; incremental data loading: the proxy-server parses the source node's AOF file and replays Redis commands; updates slot relationships; executes the remaining plan according to steps S4-S10 to complete cluster scaling; and cleans the source node data. This invention dynamically expands the existing cluster, migrates log files, and quickly transfers all data. Only incremental data requires parsing and replaying write commands from the AOF file. During the migration process, services continue to operate normally without impacting business operations.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method for rapid scaling of a Redis cluster based on a proxy architecture. Background Technology

[0002] When a business experiences rapid growth or encounters a bug, Redis capacity and traffic can easily become overloaded. Businesses often want to perform rapid, smooth, lossless, and seamless horizontal scaling without changing the IP connection address.

[0003] Often, businesses will adopt the following approach: create a new cluster, use Redis Shake to migrate data, and then the business side replaces the IP address to "expand" the cluster (new creation and change). However, this method not only has a significant impact on the business (requiring the replacement of the IP address), but also has a high probability of data loss, which can lead to business losses.

[0004] To address the current problems of Proxy-Redis clusters, such as low scaling efficiency, data loss, and noticeable scaling issues, this patent provides a highly efficient, smooth, and seamless dynamic scaling method to solve these problems. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A method for rapid scaling of a Redis cluster based on a proxy architecture includes:

[0008] S1. Precondition determination: Whether the resources are in place;

[0009] S2. Develop an execution plan: Confirm the split source node and target node, and the reallocation of slots;

[0010] S3, Parallel Control: Uses distributed locks to ensure the seriality of capacity expansion;

[0011] S4. Perform backup: Flush the memory data to disk, regenerate the rdb file, perform data backup, and reduce the amount of data in memory at the same time;

[0012] S5, RDB file transfer: Transferring RDB files from the source node to each target node;

[0013] S6, RDB file detection: Compare the MD5 values ​​of the source node and the target node;

[0014] S7. Full Data Load: Start the target node and load the required data from RDB into memory according to the allocated slots;

[0015] S8, Source Node Write Disable: The source node initiates a write disable command to ensure that no more data is added;

[0016] S9. Incremental data loading: The Proxy-Server parses the source node's AOF file and traces back the Redis commands;

[0017] S10, Update slot relationships;

[0018] S11. Execute the remaining plan according to steps S4 to S10 to complete the cluster expansion;

[0019] S12, Clean source node data.

[0020] As a preferred embodiment of the rapid scaling method for a Redis cluster based on a Proxy architecture as described in this invention, the prerequisites in S1 include the resources necessary for scaling up the Redis-Server and Proxy-Server.

[0021] As a preferred embodiment of the fast scaling method for a Redis cluster based on a proxy architecture as described in this invention, the principles to be followed in the execution plan in S2 include: a) maintaining the existing cluster; b) using a node splitting mode for horizontal scaling; and c) storing slot information in ZooKeeper.

[0022] As a preferred embodiment of the fast scaling method for a Redis cluster based on a proxy architecture as described in this invention, the step of maintaining the existing cluster specifically refers to dynamically scaling the existing cluster horizontally, and the total number of shards after scaling must be m times the current number of shards, where m>=1.

[0023] As a preferred embodiment of the fast scaling method for a Redis cluster based on a Proxy architecture as described in this invention, the horizontal scaling using the node splitting mode specifically means that when the number of nodes is changed to n times, a single node needs to migrate out (n-1) / n slots of data, while retaining only 1 / n slots of data.

[0024] As a preferred embodiment of the fast scaling method for a Redis cluster based on a proxy architecture as described in this invention, wherein: the S4 performs backup by using the bgsave command to flush the data in the Redis memory to the disk.

[0025] As a preferred embodiment of the fast scaling method for a Redis cluster based on a proxy architecture as described in this invention, step S5 uses Redis's RDB persistence function to export all data in the form of an RDB file, and then transmits the RDB file to a new Redis server.

[0026] As a preferred embodiment of the fast scaling method for a Redis cluster based on a proxy architecture as described in this invention, the write-disable command in S8 adopts a self-developed Redis command.

[0027] As a preferred embodiment of the fast scaling method for a Redis cluster based on a proxy architecture as described in this invention, wherein: updating the slot relationship in S10 specifically includes updating the slot relationship in ZooKeeper and removing the write restriction on the source node.

[0028] As a preferred embodiment of the fast scaling method for a Redis cluster based on a Proxy architecture as described in this invention, in step S12, the data cleaning of the source node is specifically processed by the Proxy-server program to filter out the data corresponding to the slots migrated out of memory.

[0029] The beneficial effects of this invention are:

[0030] Dynamic scaling is based on the existing cluster, rather than replacing all servers with new ones. This means that during the scaling process, new Redis servers can be added gradually, rather than replacing all servers at once. This ensures that business users will not experience any impact when using Redis services. Data migration is simplified: migration is performed in-situ, without the need to build new machines; migration speed is fast: logistics file migration quickly transfers all data, only incremental data requires parsing AOF replay write commands; smooth and imperceptible: during the migration process, business services continue to operate normally, with no impact on business operations. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0032] Figure 1 This is a flowchart of a fast scaling method for a Redis cluster based on a proxy architecture proposed in this invention. Detailed Implementation

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0034] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0036] Example 1

[0037] Reference Figure 1 This invention provides a method for rapid scaling of a Redis cluster based on a proxy architecture, comprising:

[0038] S1. Precondition determination: Whether the resources are in place;

[0039] S2. Develop an execution plan: Confirm the split source node and target node, and the reallocation of slots;

[0040] S3. Parallel control: Use distributed locks to ensure the seriality of expansion; this can be achieved using ZooKeeper, Database Lock, etc.

[0041] S4. Perform backup: Flush the memory data to disk, regenerate the rdb file, perform data backup, and reduce the amount of data in memory at the same time;

[0042] S5, RDB file transfer: Transferring RDB files from the source node to each target node;

[0043] S6, RDB file detection: Compare the MD5 values ​​of the source and target nodes to ensure normal file sending and receiving and data integrity;

[0044] S7. Full Data Loading: Start the target node and load the required data from RDB into memory according to the allocated slots, using self-developed Redis commands or program control.

[0045] S8, Source Node Write Disable: The source node initiates a write disable command to ensure that no more data is added;

[0046] S9. Incremental Data Loading: The Proxy-Server parses the source node's AOF file and traces back the Redis commands; it connects to the target node and executes the corresponding commands on the target node according to the allocated slots to complete the loading of incremental data.

[0047] S10, Update slot relationships; At this point, the current source node has completed the split, and the source node and target node begin providing services;

[0048] S11. Execute the remaining plan according to steps S4 to S10 to complete the cluster expansion;

[0049] S12. Clean the source node data and filter out redundant information.

[0050] Among them, the prerequisites in S1 include the resources necessary for the expansion of Redis-Server and Proxy-Server, specifically servers, JAR files, scripts, etc.

[0051] Furthermore, the principles that must be followed in the execution plan in S2 include: a) maintaining the existing cluster; b) using a node splitting mode for horizontal scaling; and c) storing slot information in ZooKeeper.

[0052] Furthermore, maintaining the existing cluster specifically means dynamically scaling the existing cluster horizontally. The total number of shards after scaling must be m times the current number of shards, where m>=1.

[0053] Furthermore, when using the node splitting mode for horizontal expansion, if the number of nodes is increased to n times, then a single node needs to migrate out (n-1) / n slots of data, while retaining only 1 / n slots of data.

[0054] Furthermore, S4 performs backups by flushing the data in Redis memory to disk using the `bgsave` command. This ensures that data is not lost even if the Redis server fails during the scaling process. Simultaneously, the disk flushing operation reduces the pressure on synchronizing subsequent incremental data.

[0055] Furthermore, S5 uses Redis's RDB persistence feature to export all data as an RDB file. Then, the RDB file is transferred to the new Redis server and loaded according to the slots using a self-developed loading command or program. This ensures that the new Redis server can quickly obtain all the data.

[0056] Furthermore, the write-disable command in S8 uses a self-developed Redis command to ensure that no more data is added.

[0057] Furthermore, updating slot relationships in S10 specifically includes updating slot relationships in ZooKeeper and removing write restrictions on the source node.

[0058] Furthermore, in S12, the source node data is cleaned through a Proxy-server program, which filters out the data corresponding to the slots migrated out of memory.

[0059] Before executing the plan, use the `bgsave` command to flush the data in Redis memory to disk. This ensures that no data is lost even if the Redis server fails during the scaling process. At the same time, flushing to disk reduces the pressure on synchronizing subsequent incremental data.

[0060] Using Redis's RDB persistence feature, the entire dataset is exported as an RDB file. Then, the RDB file is transferred to a new Redis server, where it is loaded slot by slot using a custom loading command or programmatically. This ensures that the new Redis server can quickly retrieve the full dataset.

[0061] Before incremental data synchronization, use Redis's write-disable command or function to ensure that no data is lost during the data synchronization process.

[0062] The expansion is based on the existing cluster, rather than replacing all servers with new ones. This means that during the expansion process, new Redis servers can be added gradually, rather than replacing all servers at once. This ensures that users will not experience any impact when using the Redis service.

[0063] Example 2

[0064] Reference Figure 1 This is the second embodiment of the present invention. This embodiment differs from the first embodiment in that it is used to verify its beneficial effects. The specific steps are as follows:

[0065] Step 1: Before executing the expansion plan, first perform a bgsave operation on the source node to back up the data, ensuring data security, reducing the amount of data in the AOF file, and making the subsequent synchronization process faster;

[0066] Step 2: This invention completes the transmission of all data by synchronizing RDB files. After the target node loads the RDB file, it can quickly import the data into memory and thus enter a normal working state.

[0067] Step 3: After starting the target node process, load the RDB file according to the predetermined slots using the self-developed slot loading command or program, so that only the required data enters the memory, avoiding data redundancy and waste of computing resources.

[0068] Step 4: By setting write restrictions on the source node Redis, we ensure that data will not be continuously added, thus ensuring data consistency and stability (the write operation restriction can be implemented by modifying the Redis kernel or proxy layer).

[0069] Step 5: When parsing the AOF file, this device can synchronize data to the target node according to the slot settings, further improving the accuracy and efficiency of data synchronization.

[0070] At this point, data synchronization is complete, and the node splitting process is successfully completed. This invention effectively improves the scaling efficiency and data consistency of the Proxy-Redis distributed cluster, while minimizing the impact on business operations.

[0071] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for rapid scaling of a Redis cluster based on a proxy architecture, characterized by: include: S1. Precondition determination: Whether the resources are in place; S2. Develop an execution plan: Confirm the split source node and target node, and the reallocation of slots; S3, Parallel Control: Uses distributed locks to ensure the seriality of capacity expansion; S4. Perform backup: Flush the memory data to disk, regenerate the rdb file, perform data backup, and reduce the amount of data in memory at the same time; S5, RDB file transfer: Transferring RDB files from the source node to each target node; S6, RDB file detection: Compare the MD5 values ​​of the source node and the target node; S7. Full Data Load: Start the target node and load the required data from RDB into memory according to the allocated slots; S8, Source Node Write Disable: The source node initiates a write disable command to ensure that no more data is added; S9. Incremental data loading: The Proxy-Server parses the source node's AOF file and traces back the Redis commands; S10, Update slot relationships; S11. Execute the remaining plan according to steps S4 to S10 to complete the cluster expansion; S12, Clean source node data.

2. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 1, characterized in that: The prerequisites in S1 include the resources necessary for scaling up Redis-Server and Proxy-Server.

3. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 2, characterized in that: The principles to be followed in the execution plan in S2 include: a) maintaining the existing cluster; b) using a node splitting mode for horizontal scaling; and c) storing slot information in zk.

4. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 3, characterized in that: Maintaining the existing cluster specifically means dynamically scaling the existing cluster horizontally. The total number of shards after scaling must be m times the current number of shards, where m>=1.

5. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 4, characterized in that: The horizontal expansion using the node splitting mode specifically means that if the number of nodes is increased by n times, then a single node needs to migrate out (n-1) / n slots of data, while retaining only 1 / n slots of data itself.

6. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 5, characterized in that: The S4 backup process uses the bgsave command to flush the data in Redis memory to disk.

7. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 6, characterized in that: The S5 uses Redis's RDB persistence feature to export all data as an RDB file, and then transfers the RDB file to a new Redis server.

8. The method for rapid scaling of a Redis cluster based on a proxy architecture according to claim 7, characterized in that: The write-disable command in S8 uses a self-developed Redis command.

9. A method for rapid scaling of a Redis cluster based on a proxy architecture as described in claim 8, characterized in that: The update of slot relationships in S10 specifically includes updating the slot relationships in zk and removing the write restriction on the source node.

10. A method for rapid scaling of a Redis cluster based on a proxy architecture as described in claim 9, characterized in that: In step S12, the source node data is cleaned through a Proxy-server program, which filters out the data corresponding to the slots migrated out of memory.

Images