Redis cluster federated data routing method and device, equipment, medium and product

Abstract

The invention relates to a Redis cluster federated data routing method and device, equipment, a medium and a product, and the method comprises the steps: dividing a Redis cluster federated Hash slot, and obtaining a plurality of Hash slot directories; wherein each hash slot directory comprises a plurality of hash slots; constructing a static routing table; the static routing table is used for representing a first mapping relationship between each hash slot directory and each Redis cluster under a Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and a node in each Redis cluster; and when an operation request of a client for the first node is received, searching a routing path of the first node in the Redis cluster federation based on the static routing table, and forwarding the operation request to the first node according to the routing path, so that the first node executes a corresponding operation according to the operation request. According to the method and the device, efficient data routing and seamless extension of a large-scale Redis cluster federation can be realized, so that the performance and the operation and maintenance efficiency of the Redis cluster federation are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of distributed cache systems, and particularly relates to a Redis cluster federation data routing method, device, equipment, medium and product. BACKGROUND

[0002] With the continuous expansion of Internet business scale, distributed cache systems have become the core infrastructure to support high concurrency access. Remote dictionary service (Redis) clusters are widely used in distributed cache systems due to their good performance in small and medium-sized deployments and the ability to provide efficient data access.

[0003] In related technologies, the traditional routing method of Redis cluster has inherent scalability bottlenecks. The nodes will cause the operation and maintenance complexity to increase dramatically when deployed in large scale, resulting in performance degradation. In view of this, there is an urgent need for a Redis cluster federation data routing method to manage large-scale clusters while ensuring the performance of the cluster. SUMMARY

[0004] The present disclosure provides a Redis cluster federation data routing method, device, equipment, medium and product.

[0005] According to a first aspect of the present disclosure, a Redis cluster federation data routing method is provided, the Redis cluster federation comprising a plurality of Redis clusters, each Redis cluster comprising a plurality of nodes, the method comprising: dividing the hash slots of the Redis cluster federation to obtain a plurality of hash slot directories; wherein each hash slot directory contains a plurality of hash slots; constructing a static routing table; the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster in the Redis cluster federation, and a second mapping relationship between each hash slot in each hash slot directory and the nodes in each Redis cluster; when receiving an operation request of a client to a first node, finding a routing path of the first node in the Redis cluster federation based on the static routing table, and forwarding the operation request to the first node according to the routing path, so that the first node performs corresponding operation according to the operation request.

[0006] Further, the finding of the routing path of the first node in the Redis cluster federation based on the static routing table comprises: hashing the address information of the first node to obtain a first hash value; bitwise operation is performed on the first hash value to obtain a target hash slot directory number and a target offset; The routing path is determined in the static routing table based on the target hash slot directory number and the target offset.

[0007] Further, the bitwise operation on the first hash value to obtain a target hash slot directory number and a target offset comprises: The total number of hash slots contained in the Redis cluster federation is subjected to a modulo operation based on the first hash value to obtain the target offset; The target offset is subjected to a shift processing based on a preset numerical value to obtain the target hash slot directory number.

[0008] Further, the determination of the routing path in the static routing table based on the target hash slot directory number and the target offset comprises: A Redis cluster associated with the target hash slot directory number is found based on a first mapping relationship in the static routing table to obtain a target Redis cluster; A node associated with the target offset is found in the target Redis cluster based on a second mapping relationship in the static routing table to obtain a target node, and the routing path is determined based on the location of the target node.

[0009] Further, after forwarding the operation request to the first node according to the routing path, the method further comprises: In response to a data migration instruction for a second node, a first Redis cluster associated with the second node is determined in the Redis cluster federation; wherein the first Redis cluster comprises a Redis cluster to which the second node belongs, and / or a Redis cluster to which a destination node of the second node belongs, the destination node being a terminal point for data migration of the second node; A network state indicator of the first Redis cluster is determined; wherein the network state indicator comprises at least one of the following: network delay, bandwidth, packet loss rate; A migration strategy is determined based on the network state indicator; wherein the migration strategy is used to indicate whether to realize data migration through an intermediary module, the intermediary module comprising a data transmission switch and a data transmission router in the Redis cluster federation; Data in the second node is migrated to the destination node based on the migration strategy.

[0010] Further, the determination of the migration strategy based on the network state indicator comprises: The first Redis cluster is evaluated based on the network state indicator to obtain an evaluation result; determine the migration strategy based on the evaluation result.

[0011] Further, the evaluation of the first Redis cluster based on the network status indicators includes: assigning a weight to each network status indicator respectively; performing weighted calculation on each network status indicator based on the weight to obtain the evaluation result.

[0012] Further, the determination of the migration strategy based on the evaluation result includes: if the evaluation result is greater than a preset threshold, determining the migration strategy as a first migration strategy; wherein the first migration strategy indicates data migration of the second node to the destination node; if the evaluation result is less than or equal to the preset threshold, determining the migration strategy as a second migration strategy; wherein the second migration strategy indicates data migration of the second node to the destination node through the intermediary module.

[0013] Further, the migration strategy is the first migration strategy, and the migration of the data in the second node to the destination node based on the migration strategy includes: determining a target transmission path in a network topology structure between the second node and the destination node; wherein the network topology structure is determined according to the topological relationship between the nodes in the Redis cluster to which the second node belongs and the nodes in the Redis cluster to which the destination node belongs; performing sharding processing on the data to be migrated in the second node to obtain sharded data, and transmitting the sharded data to the destination node in parallel through the target transmission path.

[0014] Further, the method further includes: acquiring change information of the network status indicators during the migration of the data in the second node to the destination node according to the migration strategy; switching the migration strategy from the first migration strategy to the second migration strategy in the case that the change information exceeds a preset change range, and migrating the data that has not been migrated to the destination node according to the second migration strategy; the first migration strategy is the data migration of the second node to the destination node; the second migration strategy is the data migration of the second node to the destination node through the intermediary module.

[0015] According to a second aspect of the present disclosure, a Redis cluster federation data routing device is provided, which includes: A first building module is used to divide the hash slots of the Redis cluster federation into multiple hash slot directories; wherein each of the hash slot directories contains multiple hash slots; A second construction module is used to construct a static routing table; the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and a node in each Redis cluster; The forwarding module is configured to, upon receiving an operation request from a client to a first node, search for a routing path of the first node in the Redis cluster federation based on the static routing table, and forward the operation request to the first node according to the routing path, so that the first node performs a corresponding operation according to the operation request.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the program.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of the present disclosure is implemented.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements the above method of the present disclosure.

[0019] The embodiments of the present disclosure provide a routing method, apparatus, device, medium and product for Redis cluster federated data. In the embodiments of the present application, first, the hash slots of the Redis cluster federation are divided to obtain multiple hash slot directories; wherein each hash slot directory contains multiple hash slots; a static routing table is constructed; the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and a node in each Redis cluster; when an operation request to a first node is received from a client, the routing path of the first node is searched in the Redis cluster federation based on the static routing table, and the operation request is forwarded to the first node according to the routing path, so that the first node performs the corresponding operation according to the operation request.

[0020] From the above description, it can be seen that the technical solution disclosed in this disclosure divides the hash slots of the Redis cluster federation into multiple hash slot directories, each containing multiple hash slots, and then constructs a static routing table containing a first mapping relationship between the hash slot directory and the Redis cluster, and a second mapping relationship between each hash slot under the hash slot directory and the node in the Redis cluster. When an operation request from the client to the first node is received, the routing path of the first node can be quickly found based on the static routing table and the operation request can be forwarded. The technical solution disclosed in this disclosure greatly simplifies the routing decision process with the help of a clear two-level mapping relationship, reduces the complexity and time consumption of routing lookup, and significantly improves the response speed of client operation requests; and this The structured hash slot directory division and static routing table design in the public technical solution can ensure that data is evenly distributed among Redis clusters and nodes under the Redis cluster federation, effectively avoiding overload of a single node or cluster, and ensuring the overall stability of the system performance. At the same time, the fixed mapping of the static routing table not only reduces the resource consumption caused by dynamic routing calculations, but also reduces the operational difficulty of routing management. It can accurately forward operation requests to the target first node, avoiding data access anomalies caused by routing errors, further improving the reliability and accuracy of Redis cluster federation data access, and providing strong support for efficient data routing of large-scale Redis cluster federation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 A flowchart of a method for routing federated data in a Redis cluster provided by an exemplary embodiment of the present disclosure; Figure 2 A schematic diagram of the structure of a Redis cluster federation provided by an exemplary embodiment of the present disclosure; Figure 3 A schematic diagram of a multi-level hash slot structure provided for an exemplary embodiment of the present disclosure; Figure 4 A schematic diagram of the memory layout of a hash slot directory provided for an exemplary embodiment of the present disclosure; Figure 5 A flowchart of a method for routing federated data in a Redis cluster provided by another exemplary embodiment of the present disclosure; Figure 6 A flowchart of a data migration method provided by an exemplary embodiment of the present disclosure; Figure 7A flowchart of determining a migration strategy based on a network status indicator is provided for an example embodiment of the present disclosure. Figure 8 A flowchart of a data migration method is provided for another example embodiment of the present disclosure. Figure 9 A functional module schematic block diagram of a routing device of Redis cluster federation data is provided for an example embodiment of the present disclosure. Figure 10 A structural block diagram of an electronic device is provided for an example embodiment of the present disclosure. Figure 11 A structural block diagram of a computer system is provided for an example embodiment of the present disclosure. Figure 12 A structural block diagram of a computer program product is provided for an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0025] The term “comprising” and variations thereof as used herein are open-ended, that is, “including but not limited to”. The term “based on” is “based, at least in part, on”. The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Related definitions are given throughout the description. It should be noted that the concepts mentioned in the present disclosure are merely used for distinguishing different devices, modules or units, and are not intended to limit the functions of the devices, modules or units.

[0026] It should be noted that the terms “one”, “multiple”, “a number of” mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as “one or more”. The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0027] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0028] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can autonomously select whether to provide the personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0029] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in a text manner in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide the personal information to the electronic device. It can be understood that the above notification and obtaining of the authorization of the user are only illustrative, and do not limit the implementation manners of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0030] In one embodiment, as shown in Figure 1 A Redis cluster federation data routing method is provided, including the following steps: Step 101, dividing the hash slots of the Redis cluster federation to obtain a plurality of hash slot directories.

[0031] Here, the execution subject can divide the hash slots of the Redis cluster federation to obtain a plurality of hash slot directories, wherein each hash slot directory contains a plurality of hash slots.

[0032] In a possible embodiment, the Redis cluster federation includes a plurality of Redis clusters, and each Redis cluster includes a plurality of nodes, as shown in Figure 2 Figure 2 ​A structure diagram of the Redis cluster federation is exemplarily shown. In the Redis cluster federation, a topmost federation cluster manager (FCM) is taken as a core component, and relies on cluster registration, cluster discovery, state monitoring, load balancing, migration scheduling, fault tolerance and recovery and other modules to realize cross-cluster data migration, load balancing and cluster life cycle management. A middle federation proxy (FP) maintains a global slot mapping and routing table based on the FCM, undertakes global management, data routing and load balancing across clusters, and synchronizes cluster discovery and slot mapping information. For the Redis cluster 1 and the Redis cluster 2 in the Redis cluster federation, a cluster interface layer (CICL) is used for internal management of each cluster. The CICL interacts with the FCM and manages data migration switches, cluster monitoring components, cluster proxies (CPs) and the like in the cluster through cluster internal management, communication, data management, state reporting, node discovery and other modules.

[0033] In addition, the Redis cluster federation can further include a data first migration module and a cluster monitoring component. The cluster monitoring component can be used to monitor the health status of the cluster, such as CPU and memory usage of nodes, request delay and the like, and make corresponding adjustments and optimizations, and can ensure the consistency of data in the cluster, including data replication, conflict resolution and consistency checking. The data first migration module can be used to manage the slot migration process in the cluster, including data migration from a source node to a target node and updating slot mapping. It should be noted that the above embodiment is only exemplarily shown as a structure diagram of the Redis cluster federation, and in actual application, the Redis cluster federation can further include more components, which are not limited herein.

[0034] In a possible embodiment, the FCM can divide the hash slots of the Redis cluster federation as an execution subject to obtain a plurality of hash slot directories. Each hash slot directory contains a plurality of hash slots. Exemplarily, in a certain Redis cluster federation, there are 16384 hash slots, which are divided according to the rule of 1024 hash slots as one hash slot directory, and finally 16 hash slot directories are obtained. Hash slot directory 0 contains hash slots 0 to 1023, hash slot directory 1 contains hash slots 1024 to 2047, and so on, and hash slot directory 15 contains hash slots 15360 to 16383. Each hash slot directory contains 1024 continuous hash slots.

[0035] In step 102, a static routing table is constructed.

[0036] Here, the execution subject divides the hash slots of the Redis cluster federation to obtain a plurality of hash slot directories, and can construct a static routing table, wherein the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and each node in each Redis cluster.

[0037] In a possible embodiment, the FCM as the execution subject can establish a global multi-level hash slot structure between the Redis clusters of the Redis cluster federation, used to coordinate the hash slots of different Redis clusters, and then configure the static routing table through the multi-level hash slot structure. The specific method is as follows: first, the FCM splits a complete hash slot into a "hash slot directory number" and a "directory number offset", as shown in Figure 3 . Figure 3 An example of a multi-level hash slot structure is shown, wherein the hash slot directory number occupies m bits, and the directory number offset occupies n bits. The hash slot directory number is used to identify the hash slot directory to which the hash slot belongs, that is, different hash slot directories can be allocated to different Redis clusters. The directory number offset is used to identify the specific position of the hash slot, used for fine routing within the Redis cluster. Then, the FCM uses a static slot mapping algorithm to allocate the hash slot to different Redis clusters, as shown in Figure 4 . Figure 4 An example of a hash slot directory memory layout is shown. The layout of the hash slot directory in the FCM adopts a static routing manner. The Start record 0 position page directory table corresponds to the ID number of the Redis cluster, and the following is sequentially occupied, finally occupying m x 4 bytes. The FCM can quickly find the IP and other information of the cluster CP through the ID number of the Redis cluster. Finally, the FCM configures the static routing table to obtain the first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and the second mapping relationship between each hash slot under each hash slot directory and each node in each Redis cluster.

[0038] For example, in a Redis cluster federation, there are Redis cluster A, Redis cluster B, and Redis cluster C. First, FCM splits each complete hash slot in the Redis cluster federation into a 4-bit "hash slot directory number" and a 12-bit "directory number offset" to build a multi-level hash slot structure. Assume that a total of 16 hash slot directories are generated. These 16 hash slot directories cover 16,384 hash slots, and each hash slot directory contains 4,096 hash slots. Then, FCM uses a static slot mapping algorithm to assign hash slot directories 0-5 to Redis cluster A, directories 6-10 to Redis cluster B, and directories 11-15 to Redis cluster C. It also uses static routing to build a hash slot directory memory layout. Starting from Start, the first 6 4-byte entries store "Redis cluster A-ID", the middle 6 4-byte entries store "Redis cluster A-ID", and the middle 6 4-byte entries store "Redis cluster A-ID". The first five store "Redis cluster B-ID" and the last five store "Redis cluster C-ID", occupying a total of 64 bytes. The Redis cluster ID can be used to quickly query information such as the CP of Redis cluster A (IP is 192.168.1.101), the CP of Redis cluster B (IP is 192.168.1.102), and the CP of Redis cluster C (IP is 192.168.1.103). Finally, FCM configures a static routing table to clarify the first mapping relationship, for example, hash slot directory 0 corresponds to cluster A, directory 6 corresponds to cluster B, etc., and the second mapping relationship, for example, under directory 0 of cluster A, hash slots 0-4095 are routed to specific nodes such as cluster A's master node A1 (192.168.1.101:6379) and slave node A2 (192.168.1.101:6380).

[0039] Step 103: When an operation request to the first node is received from the client, the routing path of the first node is searched in the Redis cluster federation based on the static routing table, and the operation request is forwarded to the first node according to the routing path, so that the first node performs the corresponding operation according to the operation request.

[0040] Here, after the execution entity constructs the static routing table, when it receives an operation request from the client to the first node, it can search the routing path of the first node in the Redis cluster federation based on the static routing table, and forward the operation request to the first node according to the routing path, so that the first node performs the corresponding operation according to the operation request.

[0041] In one possible implementation, Figure 5 As shown, searching for the routing path of the first node in the Redis cluster federation based on the static routing table includes the following steps: Step 1031: Perform a hash operation on the address information of the first node to obtain a first hash value.

[0042] Here, the execution subject can perform a hash operation on the address information of the first node to obtain a first hash value after the static routing table is constructed.

[0043] In a possible embodiment, the Redis cluster federation includes a Redis cluster A and a Redis cluster B, and the FCM has constructed a static routing table, where the Redis cluster A is responsible for a hash slot directory 0, hash slots 0-1023 under the hash slot directory 0 correspond to a master node of the Redis cluster A as a first node, and the address information of the first node is “192.168.1.10:6379”. The FCM takes the address information “192.168.1.10:6379” of the first node as input and performs a CRC16 hash operation to obtain a first hash value 0x3A5F.

[0044] In step 1032, a bit operation is performed on the first hash value to obtain a target hash slot directory number and a target offset.

[0045] Here, the execution subject can perform a bit operation on the first hash value to obtain a target hash slot directory number and a target offset after performing a hash operation on the address information of the first node to obtain a first hash value.

[0046] In a possible implementation, the bit operation on the first hash value to obtain a target hash slot directory number and a target offset includes the following steps: performing a modulo operation on the first hash value based on the total number of hash slots included in the Redis cluster federation to obtain the target offset; performing a shift operation on the target offset based on a preset value to obtain the target hash slot directory number.

[0047] Specifically, the execution subject performs an operation on the first hash value through a first formula to obtain a target hash slot directory number, where the first formula is H1=Hash(key)%(m+n)>>n, H1 is the target hash slot directory number, and Hash(key) is the first hash value. Then, the execution subject performs an operation on the first hash value through a second formula to obtain a target offset, where the second formula is H2=Hash(key)%(m+n), H2 is the target offset, m+n is the total number of hash slots of the Redis cluster federation, and n is a preset value.

[0048] In a possible embodiment, for example, it is known that the total number of hash slots of the Redis cluster federation is 16384, i.e., m+n=16384, the preset value n is 10, the execution subject performs hash operation on the address information of the first node, and obtains the first hash value Hash(key)=2560. The FCM calculates the target hash slot directory number through the first formula H1=Hash(key)%(m+n)>>n. The FCM substitutes Hash(key)=2560, m+n=16384, and n=10 into the first formula, first calculates 2560%16384=2560, and then right shifts the result by 10 bits 2560>>10=2 to obtain the target hash slot directory number H1=2. Subsequently, the FCM calculates the target offset through the second formula H2=Hash(key)%(m+n). After substituting the values, H2=2560%16384=2560 is obtained, that is, the target offset is 2560.

[0049] In step 1033, the route path is determined in the static routing table based on the target hash slot directory number and the target offset.

[0050] Here, after the execution subject performs bit operation on the first hash value to obtain the target hash slot directory number and the target offset, the route path can be determined in the static routing table based on the target hash slot directory number and the target offset.

[0051] In a possible embodiment, the route path is determined in the static routing table based on the target hash slot directory number and the target offset, including the following steps: The target Redis cluster associated with the target hash slot directory number is found based on the first mapping relationship in the static routing table, and the target Redis cluster is obtained. The node associated with the target offset is found in the target Redis cluster based on the second mapping relationship in the static routing table, the first node is obtained, and the route path is determined based on the position of the first node.

[0052] Specifically, after the execution subject performs bit operation on the first hash value to obtain the target hash slot directory number and the target offset, first, the target Redis cluster associated with the target hash slot directory number is found based on the first mapping relationship in the static routing table, and the target Redis cluster is obtained. Then, the execution subject finds the node associated with the target offset in the target Redis cluster based on the second mapping relationship in the static routing table, obtains the target node, and determines the route path based on the position of the target node.

[0053] In a possible implementation, for example, the FCM has constructed a static routing table, in which a first mapping relationship explicitly records the association between the hash slot directory and the Redis cluster, such as "hash slot directory number 2 Redis cluster A" and "hash slot directory number 5 Redis cluster B", and a second mapping relationship records the association between the offset and the node, such as "in the hash slot directory number 2 of Redis cluster A, the offset 2000-3000 corresponds to the master node A-M1". The FCM has performed hash operation on the key "order:202405" requested by the client to obtain a first hash value 2560, and has calculated the target hash slot directory number H1=2 and the target offset H2=2560 through the formula. First, the FCM determines the target Redis cluster as cluster A according to the first mapping relationship of the static routing table and the target hash slot directory number 2. Then, the FCM finds the node associated with the target offset 2560 (which belongs to the interval 2000-3000) in the node routing record of Redis cluster A based on the second mapping relationship of the static routing table, and obtains the target node as the master node A-M1. Finally, the FCM determines the complete routing path as "client->CP of cluster A->target node A-M1" by combining the CP address and the first node address of cluster A recorded in the static routing table.

[0054] In the present embodiment, first, the execution subject performs hash operation on the address information of the first node to obtain a first hash value. Then, the execution subject performs bit operation on the first hash value to obtain a target hash slot directory number and a target offset. Finally, the execution subject determines the routing path in the static routing table based on the target hash slot directory number and the target offset.

[0055] As can be seen from the above description, in the present embodiment, the execution subject performs hash operation on the first node address to obtain a unique hash value, which can provide accurate initial basis for routing positioning, and thus the execution subject can quickly obtain the target hash slot directory number and the offset through efficient bit operation, greatly shortening the routing calculation time. Finally, according to the static routing table pre-constructed by the FCM, the first node can be quickly locked and the routing path can be determined, without the need for dynamic calculation of routing information, which not only reduces the consumption of cross-cluster communication resources and avoids the routing delay caused by the increase of traditional cluster nodes, but also guarantees the stability and accuracy of routing decision, thereby effectively improving the data routing efficiency and overall reliability of the Redis cluster federation.

[0056] In a possible embodiment, after the execution subject finds the routing path of the first node in the Redis cluster federation based on the static routing table, the operation request can be forwarded to the first node according to the routing path, so that the first node performs the corresponding operation according to the operation request. The operation request includes basic key-value read-write operations, such as GET to obtain the value corresponding to the specified key, SET to store or update the key-value pair, and also covers data CRUD related operations, such as DEL to delete the specified key, EXISTS to judge whether the key exists, HGET / HSET to operate the hash type data, LPUSH / LPOP to operate the list type data, and the like. In addition, the operation request also covers basic data state query operations, such as TTL to view the expiration time of the key, TYPE to query the data type corresponding to the key, and the like. It should be noted that these operation requests need to be executed by the first node according to the instructions to perform corresponding data processing or state feedback, and after being forwarded through the routing path determined based on the static routing table, the target first node can be accurately reached to complete the operation.

[0057] The Redis cluster federation data routing method, device, equipment, medium and product provided by the embodiments of the present disclosure are provided. In the embodiments of the present disclosure, first, the hash slots of the Redis cluster federation are divided to obtain a plurality of hash slot directories; each hash slot directory contains a plurality of hash slots; a static routing table is constructed; the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and the nodes in each Redis cluster; when receiving an operation request of a client to a first node, the routing path of the first node in the Redis cluster federation is found based on the static routing table, and the operation request is forwarded to the first node according to the routing path, so that the first node performs the corresponding operation according to the operation request.

[0058] As can be known from the above description, the technical scheme of the present disclosure divides the hash slots of the Redis cluster federation into a plurality of hash slot directories each containing a plurality of hash slots, and then constructs a static routing table containing a first mapping relationship between the hash slot directories and the Redis cluster, and a second mapping relationship between each hash slot under the hash slot directory and the nodes in the Redis cluster. When receiving an operation request of a client to a first node, the routing path of the first node can be quickly found and the operation request can be forwarded based on the static routing table. The technical scheme of the present disclosure greatly simplifies the routing decision process by means of clear two-level mapping relationship, reduces the complexity and time consumption of routing lookup, and significantly improves the response speed of the client operation request. In addition, the structured hash slot directory division and the design of the static routing table in the technical scheme of the present disclosure can ensure that the data is evenly distributed among the Redis clusters and nodes in the Redis cluster federation, effectively avoid the overload of a single node or cluster, and guarantee the overall performance stability of the system. At the same time, the fixed mapping of the static routing table not only reduces the resource consumption caused by dynamic routing calculation, but also reduces the operation and maintenance difficulty of routing management, and can accurately forward the operation request to the target first node, avoid data access anomalies caused by routing errors, and further improve the reliability and accuracy of data access in the Redis cluster federation. The technical scheme provides strong support for efficient data routing of large-scale Redis cluster federation.

[0059] In one embodiment, as shown in FIG. 6, after forwarding the operation request to the first node according to the routing path, the method further includes the following steps: Figure 6 Step 601, in response to a data migration instruction to a second node, determining a first Redis cluster associated with the second node in the Redis cluster federation.

[0060] Here, after forwarding the operation request to the first node according to the routing path, the execution subject can determine a first Redis cluster associated with the second node in the Redis cluster federation in response to a data migration instruction to the second node, wherein the first Redis cluster includes a Redis cluster to which the second node belongs, and / or a Redis cluster to which a destination node of the second node belongs, and the destination node is the end point of data migration of the second node.

[0061] ​In a possible embodiment, for example, Slave node A2 of Redis cluster A receives a data migration instruction for the second node, "migrate hash slot directory 4 responsible by A2 to new node B1", the FCM determines the first Redis cluster associated with the second node based on the cluster registration information and the static routing table maintained by the FCM, for example, the FCM queries the node information of A2 in the static routing table to find that the home Redis cluster of A2 is Redis cluster A, and then it is determined that the first Redis cluster is Redis cluster A, and if the FCM queries the node information of A2 in the static routing table to find that the home Redis cluster of A2 is other Redis cluster B, then it is determined that the first Redis cluster is Redis cluster B.

[0062] In step 602, the network state indicator of the first Redis cluster is determined.

[0063] Here, after the execution subject determines the first Redis cluster associated with the second node in the Redis cluster federation, the network state indicator of the first Redis cluster can be determined, wherein the network state indicator at least includes one of the following: network delay, bandwidth, and packet loss rate.

[0064] In a possible embodiment, for example, when the execution subject determines the network state indicator of the first Redis cluster, for network delay, the round trip time (RTT) of the communication between the second node and other nodes in the first Redis cluster, the communication between the first Redis cluster and the federation cluster manager FCM and the federation proxy FP needs to be monitored, and the delay data is obtained by periodically sending probe data packets, so as to reflect the response efficiency of data transmission in the cluster and with external components; for bandwidth, the execution subject needs to count the real-time bandwidth occupation and the remaining available bandwidth of the data transmission between the nodes in the first Redis cluster, the cluster and the external, and the number of bytes transmitted per unit time is taken as the statistical dimension, so as to ensure that the bandwidth resource is sufficient in the data transmission process and avoid blocking of business requests due to bandwidth saturation; for packet loss rate, the execution subject needs to monitor the packet loss rate of the communication link related to the first Redis cluster (including the link between the nodes in the cluster, the interaction link between the cluster and the federation proxy FP and the FCM), and the packet loss rate is calculated by comparing the total amount of sent data packets with the total amount of received data packets.

[0065] In step 603, the migration strategy is determined based on the network state indicator.

[0066] Here, after the execution subject determines the network state indicator of the first Redis cluster, the migration strategy can be determined based on the network state indicator, wherein the migration strategy is used to indicate whether to realize data migration through an intermediate module, and the intermediate module includes a data transmission switch and a data transmission router in the Redis cluster federation.

[0067] In a possible embodiment, as shown in Figure 7 The determining of the migration strategy based on the network status indicators includes the following steps: In step 6031, the first Redis cluster is evaluated based on the network status indicators, and an evaluation result is obtained.

[0068] Here, after the execution subject determines the network status indicators of the first Redis cluster, the first Redis cluster can be evaluated based on the network status indicators, and an evaluation result is obtained.

[0069] In a possible embodiment, the evaluation of the first Redis cluster based on the network status indicators includes the following steps: Each network status indicator is respectively assigned a weight; The network status indicators are weighted and calculated based on the weights, and an evaluation result is obtained.

[0070] Specifically, after the execution subject determines the network status indicators of the first Redis cluster, the execution subject first assigns a weight to each network status indicator; then, the execution subject performs weighted calculation on the network status indicators based on the weights, and an evaluation result is obtained.

[0071] In a possible embodiment, the FCM respectively assigns weights to the network delay, bandwidth, and packet loss rate of the first Redis cluster. Illustratively, considering that the packet loss rate directly affects the integrity of data migration, the weight of the packet loss rate is set to 40%; the bandwidth determines the transmission efficiency of data migration, and the weight of the bandwidth is set to 35%; the network delay affects the response speed of migration, and the weight of the network delay is set to 25%. The FCM converts the actual detection values of the network status indicators into quantitative scores of 0-100 points, for example, a network delay of 50 ms corresponds to 80 points, a remaining bandwidth that meets the migration demand 1.2 times corresponds to 90 points, and a packet loss rate of 0.1% corresponds to 95 points. Then, the weighted calculation is performed based on the weights, that is, the evaluation result = (network delay score x 25%) + (bandwidth score x 35%) + (packet loss rate score x 40%). After substituting the example scores, the evaluation result is (80 x 0.25) + (90 x 0.35) + (95 x 0.4) = 20 + 31.5 + 38 = 89.5 points.

[0072] In step 6032, a migration strategy is determined based on the evaluation result.

[0073] Here, after the execution subject evaluates the first Redis cluster and obtains an evaluation result, the execution subject can determine a migration strategy based on the evaluation result.

[0074] In a possible embodiment, the determination of the migration strategy based on the evaluation result includes the following steps: If the evaluation result is greater than the preset threshold, the migration strategy is determined as the first migration strategy.

[0075] If the evaluation result is less than or equal to the preset threshold, the migration strategy is determined as the second migration strategy.

[0076] Specifically, after the evaluation subject evaluates the first Redis cluster and obtains an evaluation result, if the evaluation result is greater than the preset threshold, the migration strategy is determined as the first migration strategy, where the first migration strategy indicates that the second node directly migrates data to the destination node; if the evaluation result is less than or equal to the preset threshold, the migration strategy is determined as the second migration strategy, where the second migration strategy indicates that the second node migrates data to the destination node through an intermediary module.

[0077] In a possible embodiment, it is assumed that the federal cluster manager FCM pre-sets the preset threshold for evaluating the network state of the first Redis cluster as 80 points. In the above example, the FCM obtains the network state evaluation result of the first Redis cluster as 89.5 points through weighted calculation, which is greater than the preset threshold of 80 points. Therefore, the FCM determines the migration strategy as the first migration strategy, that is, instructs the second node to directly transmit the hash slot data to be migrated and related meta information to the destination node without the aid of the intermediary module. If the FCM calculates the evaluation result as 72 points, which is less than the preset threshold of 80 points, the FCM determines the migration strategy as the second migration strategy, that is, instructs the second node to first upload the data to be migrated to the intermediary module for temporary storage and verification, and then pushes the data to the destination node after the intermediary module confirms the data integrity, thereby reducing the impact of network instability on migration through the buffering and fault tolerance capability of the intermediary module.

[0078] Step 604: Migrate data in the second node to the destination node based on the migration strategy.

[0079] Here, after the evaluation subject determines the migration strategy based on the network state indicator, the evaluation subject can migrate data in the second node to the destination node based on the migration strategy.

[0080] In a possible embodiment, assuming that the execution subject FCM has determined the corresponding migration strategy after calculating the evaluation result based on the network state indicators of the first Redis cluster, for example, if the first migration strategy is determined, i.e., the evaluation result is greater than the preset threshold, the FCM sends a direct migration instruction to the second node, and after receiving the instruction, the second node first performs integrity check on the to-be-migrated data, such as the key-value pair data under the hash slot directory belonging to the second node, and after the check passes, the second node directly establishes a data transmission connection with the destination node, synchronizes the data to the destination node in batches, the destination node receives the data and checks batch by batch, and after confirming that there is no error, the destination node feeds back to the FCM that the receiving is completed, and the FCM updates the node mapping relationship corresponding to the part of data in the static routing table; if the second migration strategy is determined, i.e., the evaluation result is less than or equal to the preset threshold, the FCM sends a migration instruction to the second node and the intermediate module at the same time, the second node first transmits the to-be-migrated data to the intermediate module after encryption, the intermediate module stores the data and completes the check, and then the intermediate module forwards the data to the destination node after decryption according to the instruction of the FCM, the destination node feeds back to the FCM after receiving and checking the data, and the FCM updates the static routing table, thereby completing the data migration from the second node to the destination node.

[0081] In a possible embodiment, if the migration strategy is the first migration strategy, migrating the data in the second node to the destination node based on the migration strategy includes the following steps: Determining a target transmission path in the network topology structure between the second node and the destination node; wherein the network topology structure is determined according to the topological relationship between the nodes in the Redis cluster to which the second node belongs and the nodes in the Redis cluster to which the destination node belongs; Sharding the to-be-migrated data in the second node to obtain sharded data, and transmitting the sharded data to the destination node in parallel through the target transmission path.

[0082] In a possible embodiment, for example, it is assumed that the Redis cluster to which the second node belongs is cluster X, the cluster X includes nodes X1, X2 and X3, X2 is the second node, the core network device of the cluster X is switch SX, the IP address of the SX is 192.168.10.1, the Redis cluster to which the destination node belongs is cluster Y, the cluster Y includes nodes Y1, Y2 and Y3, Y1 is the destination node, the core network device of the cluster Y is switch SY, the IP address of the SY is 192.168.20.1, the network topology between the Redis cluster to which the second node belongs and the Redis cluster to which the destination node belongs clearly records the connection relationship between the nodes and the core devices and the connection relationship between the core devices, for example, all the nodes of the cluster X are directly connected to the switch SX, all the nodes of the cluster Y are directly connected to the switch SY, there are two direct communication links between the SX and the SY, which are link A and link B, because the migration strategy is the first migration strategy, the execution subject FCM first detects the real-time network state of the two links, the detection result shows that the network delay of the link A is 20 ms, the available bandwidth is 1.2 Gbps, and the packet loss rate is 0.02%, the network delay of the link B is 65 ms, the available bandwidth is 550 Mbps, and the packet loss rate is 0.2%, in combination with the demand for transmission efficiency and stability of the direct migration in the first migration strategy, the execution subject determines that the link A is the target transmission path, and the specific direction of the path is the second node X2→the switch SX→the link A→the switch SY→the destination node Y1.

[0083] Subsequently, the execution subject sends a data sharding instruction to the second node X2, which stores the data to be migrated. For example, the total amount of data is 10 GB, corresponding to all key-value pair data under the hash slot directory 8 in the Redis cluster federation, including user order cache data and commodity information cache data. The second node X2 shards the data to be migrated according to the specification of 2 GB per slice, and finally obtains 5 shard data, named shard 1, shard 2, shard 3, shard 4, and shard 5 respectively. At the same time, a unique SHA256 checksum is generated for each shard data for integrity verification. After the sharding is completed, the second node X2 starts 5 independent transmission processes through the target transmission path link A, each transmission process corresponding to a shard data, and synchronously sends data to the destination node Y1. During transmission, each process will feed back the transmission progress to the execution subject in real time, for example, "shard 2 has completed 60% of transmission". When the destination node Y1 receives the shard data, it will verify the data integrity with the corresponding SHA256 checksum after receiving each shard. After the verification is passed, the destination node Y1 returns the "shard reception complete" confirmation information to the second node X2 through the target transmission path, and continues to receive the next shard. When all 5 shards are transmitted and the verification is passed, the destination node Y1 sends a notification of "all shard reception complete" to the execution subject, and the FCM records the transmission result, completing the parallel transmission of the shard data under the first migration strategy.

[0084] In this embodiment, first, the execution subject responds to the data migration instruction of the second node to determine the first Redis cluster associated with the second node in the Redis cluster federation; wherein the first Redis cluster includes the Redis cluster to which the second node belongs, and / or the Redis cluster to which the destination node of the second node belongs; then, the execution subject determines the network state index of the first Redis cluster; wherein the network state index at least includes: network delay, bandwidth, packet loss rate; thereafter, the execution subject determines the migration strategy based on the network state index; wherein the migration strategy is used to indicate whether to realize data migration through the intermediary module; finally, the execution subject migrates the data in the second node to the destination node based on the migration strategy.

[0085] As can be known from the above description, in the embodiment, the execution subject first locks the first Redis cluster of the associated second node, explicitly specifies the cluster range involved in the migration, avoids cross-cluster coordination confusion, and clearly demarcates the boundary for subsequent operations; then, by monitoring network delay, bandwidth, packet loss rate and the like, the execution subject provides a quantitative basis for the migration strategy and solves the adaptation problem caused by subjective decision-making; then, the execution subject determines whether to use the migration strategy of the mediation module according to the indexes, which can flexibly adapt to different network conditions, save the mediation module to improve efficiency when the network is good, and guarantee transmission stability through the mediation module when the network is poor; finally, the migration is completed according to the strategy, which can shorten the business interruption time and ensure data integrity, thereby improving the flexibility and reliability of the Redis cluster federation data migration as a whole, effectively supporting the stable operation of the Redis cluster in the scenarios of expansion, load balancing and the like, and reducing the impact of migration on business services.

[0086] In a possible embodiment, as shown in Figure 8 the routing method of the Redis cluster federation data further includes the following steps: Step 801, in the process of migrating the data in the second node to the destination node according to the migration strategy, the change information of the network state index is obtained.

[0087] Here, in the process of migrating the data in the second node to the destination node according to the migration strategy, the FCM can obtain the change information of the network state index.

[0088] In a possible embodiment, for example, the Redis cluster instructs the second node A2 to start migrating data to the destination node B1 according to the first migration strategy, for example, the total amount of data to be migrated is 15 GB, which is initially divided into 7 shards of 2 GB each, and is transmitted in parallel through the target transmission path (cluster A switch → dedicated direct link → cluster B switch). In the migration process, the FCM starts a real-time monitoring mechanism, collects the network state index of the first Redis cluster every 5 seconds and records the change information: in the initial stage of migration (the first 15 minutes), the network delay is stable at about 35 ms, the available bandwidth is maintained at 1.2 Gbps, and the packet loss rate is 0.05%. Starting from the 16th minute, the network delay gradually increases, reaching 110 ms at the 16th minute and 10 seconds, and then continuously maintaining between 105 ms and 120 ms. At the same time, the available bandwidth starts to decrease, reaching 480 Mbps at the 16th minute and 15 seconds, and then maintaining between 450 Mbps and 490 Mbps. The packet loss rate also increases from 0.05% to 1.2%, and stabilizes between 1.2% and 1.5% after the 16th minute and 20 seconds. The execution subject obtains the change information of these network state indexes in real time.

[0089] Step 802 : When it is determined that the change information exceeds the preset change range, the migration strategy is switched from the first migration strategy to the second migration strategy, and the unmigrated data is migrated to the destination node according to the second migration strategy.

[0090] Here, after obtaining the change information of the network status indicator, the execution entity can switch the migration strategy from the first migration strategy to the second migration strategy if it determines that the change information exceeds the preset change range, and migrate the unmigrated data to the destination node according to the second migration strategy, wherein the first migration strategy is to migrate data from the second node to the destination node; the second migration strategy is to migrate the data in the second node to the destination node through the intermediary module.

[0091] In a possible embodiment, continuing with the above example, FCM obtains change information of network status indicators. For example, from the 16th minute and 20 seconds, the network delay has exceeded 100ms for 10 seconds, the available bandwidth has been less than 500Mbps for 10 seconds, and the packet loss rate has been greater than 1% for 10 seconds. FCM determines that these change information exceeds the preset change range. Then, FCM sends a policy switching instruction to the second node A2, the destination node B1 and the intermediary module to switch the migration strategy from the first migration strategy to the second migration strategy, that is, migrate the data through the intermediary module. At this time, the second node A2 has completed the migration of shards 1 to shard 4 (a total of 8GB of data), and the remaining shard 5 Up to shard 7 (7GB of data in total) has not been migrated. According to the second migration strategy, the second node A2 stops transmitting data directly to the destination node B1, and instead encrypts shards 5 to 7 in sequence and transmits them to the intermediary module. After receiving each shard, the intermediary module first performs an integrity check. After the check passes, it stores it in the local cache. After all three unmigrated shards are received, the intermediary module decrypts shards 5 to 7 according to the instructions and forwards them to the destination node B1 piece by piece. The destination node B1 verifies each shard again after receiving it, and after confirming that it is correct, it feedbacks the reception status to the execution entity until all unmigrated data transmission is completed. The FCM records the migration results, completing the entire data migration process.

[0092] In this embodiment, in the process of migrating data in the second node to the destination node according to the migration strategy, change information of the network status indicator is obtained. Then, when it is determined that the change information exceeds the preset change range, the migration strategy is switched from the first migration strategy to the second migration strategy, and the unmigrated data is migrated to the destination node according to the second migration strategy.

[0093] As can be known from the above description, in the present embodiment, the execution subject acquires the change of the network state index in real time in the data migration process, can timely capture conditions such as network delay rising, bandwidth decreasing, and packet loss rate rising, and avoid the first migration strategy adapted to the original network from causing data transmission interruption, packet loss, or efficiency decrease; in the case that the change information is determined to be out of the preset change range, the execution subject actively switches the migration strategy from the first migration strategy to the second migration strategy, makes up for the transmission defects of the deteriorated network state by means of the buffering and fault tolerance capability of the intermediary module, ensures the data not yet migrated to continue migrating in a relatively stable path, guarantees the integrity and continuity of data migration, this kind of dynamic switching mechanism breaks the limitation of traditional single strategy throughout the whole process, makes the migration flexibly adapt to network fluctuation; at the same time, the non-migrated data is processed in order after the strategy switching, reduces the migration retry or retransmission caused by network problems, shortens the overall migration time consumption, reduces the influence on the Redis cluster service to which the second node and the destination node belong, and further supports the stable operation of the cluster federation.

[0094] In a possible embodiment, a binary communication protocol (RFCP protocol) is also provided, which can replace the traditional RESP protocol as a core component of the Redis cluster federation system. The implementation steps are as follows: first, the protocol positioning and core value are determined, that is, the data exchange standard between system components is provided to support cross-cluster operations, guarantee data security, system reliability and performance optimization, and improve compatibility, maintainability and scalability through standardized definition; then a unified binary data format and structure are defined, the protocol includes a fixed 16-byte message header and a variable message body, the message header includes magic number, version, message type, message ID, message length, checksum, source / destination node type and ID, etc. Metadata, and the message is divided into different secondary types, including broadcast message with type identifier 0x1, cluster registration message with type identifier 0x2, etc. To ensure data exchange standardization between different components such as federation agent and cluster agent and different versions of Redis instances, reduce parsing errors and data inconsistency risks; then, by virtue of the advantages of the protocol, such as binary compression to reduce bandwidth occupation, efficient parsing to reduce CPU overhead, support for complex operations such as expansion and contraction, embedded encryption and verification to improve security, a protocol processing module based on TCP is constructed, which includes a binary parsing module responsible for decoding and instruction parsing, a data compression module responsible for reducing transmission data volume, a security verification module responsible for preventing data tampering and interception, to guarantee data transmission reliability and stability; finally, a general message processing module is implemented, which is used for protocol sending and receiving by components such as federation cluster manager, the module first creates, binds and listens to server Socket through Socket management module and epoll event loop module, and then manages client Socket registration and readable, writable, error and other event processing, then the received data is parsed and processed by the RFCP protocol analysis and processing module, the messages such as node state update and data synchronization request are classified and processed according to the message type, and the corresponding operations are performed, then the response message is constructed according to the business logic and sent to the client, at the same time, the error handling and resource management module handles abnormal situations to ensure correct resource release and system stability, and efficient, safe and reliable communication and management across clusters are realized.

[0095] In a possible embodiment, a method for implementing the expansion and contraction mode of the Redis cluster federation is also provided, and the specific steps are as follows: when the Redis cluster is expanded, the Redis cluster federation automatically adds new nodes after sensing the load increase, and migrates part of the data to the new nodes by redistributing the data shards and adopting the incremental migration strategy, while processing multiple shards in parallel to improve efficiency; when the Redis cluster is contracted, the Redis cluster federation automatically identifies idle or low-load nodes and reallocates the data shards it carries to the remaining nodes, reducing redundant resources and ensuring uninterrupted service; when the Redis clusters are expanded, the Redis cluster federation migrates data from the source cluster to the new target cluster, supports cross-cluster replication of data shards, and uses the global view of the federation architecture to ensure data integrity. According to the balanced distribution, after the expansion is completed, the new cluster joins the federated network to support cross-cluster load balancing and data synchronization; when scaling down between Redis clusters, the Redis cluster federation migrates the data of the low-load cluster to other clusters through incremental synchronization technology to ensure data integrity and consistency; during the entire scaling process, the Redis cluster federation reduces the migration amount and time by capturing the incremental changes in the Redis cluster. Shard migration uses multi-threaded processing to avoid performance bottlenecks. At the same time, it relies on data version control and cross-cluster multi-level verification, such as data hash value comparison and data version checking, to ensure data consistency and support online scaling. During the migration, the cluster continues to process read and write requests, and automatically switches to the new node or target cluster after the migration is completed, making scaling transparent to external services.

[0096] From the above description, it can be seen that the technical solution disclosed in the present invention solves the limitations of traditional Redis clusters in scalability, network communication, data consistency and management complexity, and on this basis provides stronger scalability, more flexible data management, lower cross-regional access latency and higher system reliability; at the same time, the technical solution disclosed in the present invention achieves optimization through a multi-level hash slot mechanism, effectively improving the routing efficiency of federal agents and cluster agents, further enhancing the reliability and availability of the system, and can also solve the challenges encountered in the process of large-scale cluster expansion and reduction, and can dynamically select direct communication or path optimization transmission mode according to network conditions, significantly improving the efficiency and stability of data migration.

[0097] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a routing device for Redis cluster federated data. The routing device for Redis cluster federated data may be a server or a chip applied to a server. Figure 9 This is a schematic block diagram of the functional modules of a routing device for Redis cluster federated data provided by an exemplary embodiment of the present disclosure. Figure 9 As shown, the routing device of the Redis cluster federated data includes: The first construction module 901 is configured to divide hash slots of a Redis cluster federation to obtain a plurality of hash slot directories; each of the hash slot directories contains a plurality of hash slots. The second construction module 902 is configured to construct a static routing table; the static routing table is used to represent a first mapping relationship between each of the hash slot directories and each of Redis clusters under the Redis cluster federation, and a second mapping relationship between each hash slot under each of the hash slot directories and a node in each of the Redis clusters. The forwarding module 903 is configured to, when receiving an operation request of a client to a first node, find a routing path of the first node in the Redis cluster federation based on the static routing table, and forward the operation request to the first node according to the routing path, so that the first node performs a corresponding operation according to the operation request.

[0098] In one embodiment, the forwarding module 903 includes: The first calculation unit is configured to perform a hash operation on address information of the first node to obtain a first hash value. The second calculation unit is configured to perform a bit operation on the first hash value to obtain a target hash slot directory number and a target offset. The first determination unit is configured to determine the routing path in the static routing table based on the target hash slot directory number and the target offset.

[0099] In one embodiment, the forwarding module 903 includes: The third calculation unit is configured to perform a modulo operation on a total number of hash slots contained in the Redis cluster federation based on the first hash value to obtain the target offset. The fourth calculation unit is configured to perform a shift processing on the target offset based on a preset value to obtain the target hash slot directory number.

[0100] In one embodiment, the forwarding module 903 includes: The first finding unit is configured to find a Redis cluster associated with the target hash slot directory number based on the first mapping relationship in the static routing table to obtain a target Redis cluster. The second finding unit is configured to find a node associated with the target offset in the target Redis cluster based on the second mapping relationship in the static routing table to obtain the target node, and determine the routing path based on a position of the target node.

[0101] In one embodiment, the apparatus further includes: a response module, configured to determine, in response to a data migration instruction for a second node, a first Redis cluster associated with the second node in the Redis cluster federation; wherein the first Redis cluster comprises a Redis cluster to which the second node belongs, and / or a Redis cluster to which a destination node of the second node belongs, the destination node being a terminal point for data migration of the second node; a first determination module, configured to determine a network state indicator of the first Redis cluster; wherein the network state indicator comprises at least one of the following: network delay, bandwidth, and packet loss rate; a second determination module, configured to determine a migration strategy based on the network state indicator; wherein the migration strategy is used to indicate whether to implement data migration through an intermediary module, the intermediary module comprising a data transmission switch and a data transmission router in the Redis cluster federation; a first migration module, configured to migrate data in the second node to the destination node based on the migration strategy.

[0102] In an embodiment, the second determination module comprises: an evaluation unit, configured to evaluate the first Redis cluster based on the network state indicator to obtain an evaluation result; a second determination unit, configured to determine the migration strategy based on the evaluation result.

[0103] In an embodiment, the second determination module comprises: an allocation unit, configured to allocate a weight to each network state indicator respectively; a fifth calculation unit, configured to perform weighted calculation on each network state indicator based on the weight to obtain the evaluation result.

[0104] In an embodiment, the second determination module comprises: a third determination unit, configured to determine the migration strategy as a first migration strategy if the evaluation result is greater than a preset threshold; wherein the first migration strategy is used to instruct the second node to perform data migration to the destination node; a fourth determination unit, configured to determine the migration strategy as a second migration strategy if the evaluation result is less than or equal to the preset threshold; wherein the second migration strategy is used to instruct to migrate data in the second node to the destination node through the intermediary module.

[0105] In an embodiment, the first migration module comprises: A fifth determining unit is configured to determine a target transmission path in a network topology between the second node and the destination node, wherein the network topology is determined according to a topological relationship between nodes in a Redis cluster to which the second node belongs and nodes in a Redis cluster to which the destination node belongs. A transmission unit is configured to perform sharding processing on the data to be migrated in the second node to obtain sharded data, and perform parallel transmission of the sharded data to the destination node through the target transmission path.

[0106] In an embodiment, the apparatus further includes: An obtaining module is configured to obtain change information of the network state indicator in a process of migrating data in the second node to the destination node according to the migration strategy. A second migration module is configured to switch the migration strategy from a first migration strategy to a second migration strategy in a case where the change information is determined to be out of a preset change range, and migrate data that has not been migrated to the destination node according to the second migration strategy; the first migration strategy is that the second node migrates data to the destination node; and the second migration strategy is that the intermediary module is used to migrate data in the second node to the destination node.

[0107] Figure 10 A structural schematic diagram of an electronic device provided by an example embodiment of the present disclosure is shown in FIG. 1. Figure 10 As shown in FIG. 1, the electronic device 1000 includes at least one processor 1001 and a memory 1002 coupled to the processor 1001, and the processor 1001 can execute corresponding steps in the above-mentioned method disclosed by the embodiments of the present disclosure.

[0108] The processor 1001 can also be referred to as a central processing unit (CPU), which can be an integrated circuit chip that has the processing capability of signals. Each step in the method disclosed in the embodiments of the present disclosure can be completed by the integrated logic circuit of hardware or the instructions in the form of software in the processor 1001. The processor 1001 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as hardware code processing for execution, or executed by a combination of hardware and software modules in the code processing. The software module can be located in the memory 1002, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The processor 1001 reads the information in the memory 1002 and completes the steps of the above method in combination with the hardware thereof.

[0109] In addition, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can be loaded from a storage medium or a network to a computer system with a dedicated hardware structure, such as Figure 11 The computer system 1100 shown is intended to represent various forms of digital electronic computer equipment, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computer equipment. The electronic equipment can also represent various forms of mobile devices, such as a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are merely examples, and are not intended to limit the implementations described and / or claimed in this document. Figure 11 A block diagram of a computer system for an example embodiment of the present disclosure is provided.

[0110] The computer system 1100 is intended to represent various forms of digital electronic computer equipment, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computer equipment. The electronic equipment can also represent various forms of mobile devices, such as a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are merely examples, and are not intended to limit the implementations described and / or claimed in this document.

[0111] As shown in FIG. 11, the computer system 1100 includes one or more processors, such as a processor 1104. The processor 1104 is connected to a communication infrastructure 1102 of the computer system 1100. Figure 11As shown, the computer system 1100 includes a computing unit 1101 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from the storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the computer system 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0112] A plurality of components in the computer system 1100 are connected to the I / O interface 1105, including an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device that can input information to the computer system 1100, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 1107 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1109 allows the computer system 1100 to exchange information / data with other devices through a network such as the Internet, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0113] The computing unit 1101 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above. For example, in some embodiments, the above-described methods disclosed by embodiments of the present disclosure can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1000 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the above-described methods disclosed by embodiments of the present disclosure by any other appropriate means, for example, by means of firmware.

[0114] The embodiment of the present disclosure further provides a computer readable storage medium, wherein when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above method disclosed by the embodiment of the present disclosure.

[0115] The computer readable storage medium in the embodiment of the present disclosure can be a tangible medium, which can contain or store programs for use by or in connection with an instruction execution system, apparatus or device. The above computer readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specifically, the above computer readable storage medium can include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0116] The above computer readable medium can be included in the above electronic device; or can exist separately without being assembled into the electronic device.

[0117] Figure 12 A computer program product 1200 is provided for an exemplary embodiment of the present disclosure, and the computer program product 1200 includes a computer program 1201, wherein the computer program 1201 is executed by a processor to implement the above method disclosed by the embodiment of the present disclosure.

[0118] In the embodiments of the present disclosure, computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of the same, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" programming language or similar programming languages. The program code can be executed entirely on a user computer, partially on a user computer, as an independent software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer.

[0119] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0120] The modules, components or units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of the module, component or unit does not constitute a limitation on the module, component or unit itself.

[0121] The functions described in this specification can be implemented in part or in whole by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0122] The above description is merely illustrative of the embodiments of the present disclosure and the principles of the technology involved. It is understood that the disclosure scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features thereof without departing from the above disclosure concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0123] Although some specific embodiments of the present disclosure have been described in detail by way of examples, one skilled in the art should understand that the above examples are only for illustration, and are not intended to limit the scope of the present disclosure. One skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for routing data in a Redis cluster federation, wherein the Redis cluster federation includes multiple Redis clusters, each of which includes multiple nodes, characterized in that: include: Dividing the hash slots of the Redis cluster federation to obtain multiple hash slot directories; wherein each of the hash slot directories contains multiple hash slots; Constructing a static routing table; the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and a node in each Redis cluster; When an operation request from a client to a first node is received, a routing path of the first node is searched in the Redis cluster federation based on the static routing table, and the operation request is forwarded to the first node according to the routing path, so that the first node performs a corresponding operation according to the operation request.

2. The method according to claim 1, characterized in that The searching for a routing path of the first node in the Redis cluster federation based on the static routing table includes: Performing a hash operation on the address information of the first node to obtain a first hash value; Performing a bitwise operation on the first hash value to obtain a target hash slot directory number and a target offset; The routing path is determined in the static routing table based on the target hash slot directory number and the target offset.

3. The method according to claim 2, characterized in that The performing a bitwise operation on the first hash value to obtain a target hash slot directory number and a target offset includes: Performing a modulo operation on the total number of hash slots included in the Redis cluster federation based on the first hash value to obtain the target offset; The target offset is shifted based on a preset value to obtain the target hash slot directory number.

4. The method according to claim 3, characterized in that Determining the routing path in the static routing table based on the target hash slot directory number and the target offset includes: Based on the first mapping relationship in the static routing table, searching for a Redis cluster associated with the target hash slot directory number to obtain a target Redis cluster; Based on the second mapping relationship in the static routing table, a node associated with the target offset is searched in the target Redis cluster to obtain the target node, and the routing path is determined based on the location of the target node.

5. The method according to claim 1, wherein After forwarding the operation request to the first node according to the routing path, the method further includes: In response to a data migration instruction for the second node, determining a first Redis cluster associated with the second node in the Redis cluster federation; wherein the first Redis cluster includes the Redis cluster to which the second node belongs and / or the Redis cluster to which a destination node of the second node belongs, the destination node being the end point of data migration for the second node; Determine a network status indicator of the first Redis cluster; wherein the network status indicator includes at least one of the following: network delay, bandwidth, and packet loss rate; Determining a migration strategy based on the network status indicator; wherein the migration strategy is used to indicate whether to implement data migration through an intermediary module, the intermediary module including a data transmission switch and a data transmission router in the Redis cluster federation; The data in the second node is migrated to the destination node based on the migration policy.

6. The method according to claim 5, characterized in that The determining of the migration strategy based on the network status indicator includes: Evaluate the first Redis cluster based on the network status indicator to obtain an evaluation result; The migration strategy is determined based on the evaluation result.

7. The method according to claim 6, characterized in that The evaluating the first Redis cluster based on the network status indicator to obtain an evaluation result includes: Assigning a weight to each of the network status indicators; A weighted calculation is performed on each of the network status indicators based on the weights to obtain the evaluation result.

8. The method according to claim 6, characterized in that Determining the migration strategy based on the evaluation result includes: If the evaluation result is greater than a preset threshold, the migration strategy is determined as a first migration strategy; wherein the first migration strategy is to instruct the second node to migrate data to the destination node; If the evaluation result is less than or equal to a preset threshold, the migration strategy is determined as a second migration strategy; wherein the second migration strategy is to instruct to migrate the data in the second node to the destination node through the intermediary module.

9. The method according to claim 8, characterized in that The migration strategy is a first migration strategy, and migrating the data in the second node to the destination node based on the migration strategy includes: Determining a target transmission path in a network topology between the second node and the destination node; wherein the network topology is determined based on a topological relationship between nodes in a Redis cluster to which the second node belongs and nodes in a Redis cluster to which the destination node belongs; The data to be migrated in the second node is fragmented to obtain fragmented data, and the fragmented data is transmitted in parallel to the destination node through the target transmission path.

10. The method according to claim 5, characterized in that The method further comprises: In the process of migrating the data in the second node to the destination node according to the migration strategy, obtaining change information of the network status indicator; When it is determined that the change information exceeds the preset change range, the migration strategy is switched from the first migration strategy to the second migration strategy, and the unmigrated data is migrated to the destination node according to the second migration strategy; the first migration strategy is to migrate data from the second node to the destination node; the second migration strategy is to migrate the data in the second node to the destination node through the intermediary module.

11. A routing device for Redis cluster federated data, characterized in that: include: A first building module is used to divide the hash slots of the Redis cluster federation into multiple hash slot directories; wherein each of the hash slot directories contains multiple hash slots; A second construction module is used to construct a static routing table; the static routing table is used to represent a first mapping relationship between each hash slot directory and each Redis cluster under the Redis cluster federation, and a second mapping relationship between each hash slot under each hash slot directory and a node in each Redis cluster; The forwarding module is configured to, upon receiving an operation request from a client to a first node, search for a routing path of the first node in the Redis cluster federation based on the static routing table, and forward the operation request to the first node according to the routing path, so that the first node performs a corresponding operation according to the operation request.

12. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images