Redis cluster high-availability deployment method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a Redis cluster high-availability deployment method, device, equipment and storage medium. The method comprises: obtaining an undeployed master node in the Redis cluster as a target master node; determining a host that meets a set condition as a first target host, wherein the set condition is that the proportion of the number of deployed master nodes in the Redis cluster does not exceed a set threshold; deploying the target master node on the first target host, and deploying a slave node corresponding to the target master node on a second target host, wherein the second target host is different from the first target host. The deployment method provided by the embodiments of the present application can guarantee the high availability of the Redis cluster and improve the resource utilization of the host in the scenarios of creating the Redis cluster, fault restart, horizontal expansion and the like.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of database deployment, and particularly relate to a Redis cluster high-availability deployment method, device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of the Internet, big data applications and cloud computing, in order to meet the high concurrency, high performance and high scalability requirements of application services, the resource utilization and cluster availability need to be considered when deploying and maintaining a Remote Dictionary Server (Redis) cluster.

[0003] The existing Kubernetes default scheduler can deploy nodes in the Redis cluster to a host with sufficient resources and meet the requirements of Pod affinity and anti-affinity. However, the high-availability requirement of the Redis cluster in the actual production environment is more stringent. When more than half of the master nodes are deployed on the same host, if the host is down and triggers the Redis master node election, at this time, since more than half of the master nodes are unavailable, the election will fail, further causing the Redis cluster to be unavailable, thereby causing serious impact. Similarly, if a pair of master-slave nodes fails at the same time, the cluster will also be unavailable due to the failure of the election. Especially when the number of hosts is small and the size of the Redis cluster shards is large, the Kubernetes default scheduler cannot meet this higher availability requirement. SUMMARY

[0004] Embodiments of the present application provide a Redis cluster high-availability deployment method, device, equipment and storage medium, which can realize the deployment of Redis cluster nodes, improve the reliability of deployment, and especially in the case of a small number of hosts and a large size of Redis cluster shards, the availability of the Redis cluster can still be guaranteed.

[0005] In a first aspect, embodiments of the present application provide a Redis cluster high-availability deployment method, the Redis cluster includes a plurality of shards, and each shard includes a master node and a slave node; the method includes:

[0006] obtaining an undeployed master node in the Redis cluster as a target master node;

[0007] determining a host that meets a set condition as a first target host; wherein the set condition is that the proportion of the number of deployed master nodes in the Redis cluster does not exceed a set threshold;

[0008] deploying the target master node on the first target host.

[0009] In a second aspect, the embodiments of the present application further provide a Redis cluster high-availability deployment device, the Redis cluster comprising a plurality of shards, and each shard comprising a master node and a slave node; the method comprising:

[0010] a target master node acquisition module, configured to acquire one master node not deployed in the Redis cluster as a target master node;

[0011] a first target host determination module, configured to determine a host satisfying a set condition as a first target host; wherein the set condition is that a proportion of the number of master nodes of the Redis cluster deployed does not exceed a set threshold;

[0012] a node deployment module, configured to deploy the target master node to the first target host.

[0013] In a third aspect, the embodiments of the present application further provide an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory in communication with the at least one processor; wherein

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the Redis cluster high-availability deployment method according to the embodiments of the present application.

[0017] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium storing computer instructions, and the computer instructions are used to enable a processor to implement the Redis cluster high-availability deployment method according to the embodiments of the present application when executed.

[0018] The embodiments of the present application disclose a Redis cluster high-availability deployment method, device, equipment and storage medium. One master node not deployed in a Redis cluster is acquired as a target master node; a host satisfying a set condition is determined as a first target host; wherein the set condition is that a proportion of the number of master nodes of the Redis cluster deployed does not exceed a set threshold; and the target master node is deployed to the first target host. The deployment method provided by the embodiments of the present application can guarantee the high availability of the Redis cluster and improve the resource utilization of the host in the scenarios of creating the Redis cluster, fault restart and horizontal expansion. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1is a flow chart of a Redis cluster high-availability deployment method in the first embodiment of the present application;

[0020] Figure 2 is an example diagram of a Redis cluster in the first embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of a Redis cluster high-availability deployment device in the second embodiment of the present application;

[0022] Figure 4 is a structural schematic diagram of an electronic device in the third embodiment of the present application. DETAILED DESCRIPTION

[0023] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0024] In the present embodiment, the existing open source K8s default scheduler is extended based on the Kubernetes scheduling framework, and a custom scheduling logic is implemented at the Filter, PreBind and PostBind extension points. The master nodes in the Redis cluster are scheduled through the K8s default scheduler and the custom scheduling logic, and the slave nodes are scheduled through the K8s default scheduler. The Redis cluster high-availability deployment method of the present embodiment can be applied to any scenario that requires deployment of a Redis cluster, such as system failure restart, Redis cluster horizontal expansion, etc.

[0025] Embodiment I

[0026] Figure 1 A flow chart of a Redis cluster high-availability deployment method provided in the first embodiment of the present application, the present embodiment can be applied to the case of deploying a Redis cluster on a host machine. The method can be executed by a Redis cluster high-availability deployment device, which can be implemented in the form of software and / or hardware, and can be implemented by an electronic device, which can be a mobile terminal, a PC terminal or a server, etc. As shown in the figure, the method specifically includes the following steps: Figure 1

[0027] S110, obtaining an undeployed master node in the Redis cluster as a target master node.

[0028] In the present embodiment, the master nodes and slave nodes in the shards can be named. For example,​Figure 2 is an example diagram of the Redis cluster in this embodiment, as Figure 2 shown, the Redis cluster includes three shards, namely shard 0, shard 1 and shard 2, shard 0 contains two nodes named pod-0-0 and pod-0-1 respectively, shard 1 contains two nodes named pod-1-0 and pod-1-1 respectively, and shard 2 contains two nodes named pod-2-0 and pod-2-1 respectively. The master-slave relationship of the two nodes in the shard is set by the user. If one of the nodes is selected as the master node, the other node is the slave node.

[0029] Optionally, the way to obtain an undeployed master node in the Redis cluster can be: obtaining a pre-configured regular expression; determining a master node from the undeployed nodes according to the regular expression.

[0030] The regular expression can be understood as a pre-set rule for specifying which node in the shard is the master node. For example, it can be specified that the node with “0” at the end of the name is the master node, then Figure 2 the master nodes in the Redis cluster in are pod-0-0, pod-1-0 and pod-2-0, and the slave nodes are pod-0-1, pod-1-1 and pod-2-1.

[0031] The undeployed node can be understood as a node in the Redis cluster that has not been deployed to the host or has not been bound to the host. Specifically, the pre-configured regular expression is matched with the node that has not been deployed to the host in the Redis cluster, and one of the matched nodes is determined as the target master node.

[0032] S120, determine a host that meets the set condition as the first target host.

[0033] The set condition is that the proportion of the number of deployed master nodes in the Redis cluster does not exceed the set threshold. The set threshold can be any value between 45%-55%, and in this embodiment, it can be set to 50%. The host is a worker node in the K8S cluster, and each host can be assigned some workloads (docker containers) for deploying nodes in the Redis cluster.

[0034] In this embodiment, the process of determining a host that meets the set condition as the first target host can be: obtaining the total number of master nodes included in the Redis cluster; for each host, obtaining the number of deployed master nodes of the host; determining the proportion according to the number of deployed master nodes and the total number of master nodes; if the proportion does not exceed the set threshold, the host meets the set condition.

[0035] The number of host machines that have deployed master nodes can be understood as the number of host machines that have deployed master nodes of the same Redis cluster. The process of determining the number ratio can be: dividing the number of deployed master nodes by the total number of master nodes of the Redis cluster to obtain the number ratio of deployed master nodes. If the number ratio of deployed master nodes is greater than or equal to the set threshold, the host machine does not meet the set condition, and if the number of deployed master nodes is less than (i.e., does not exceed) the set threshold, the host machine meets the set condition. In this embodiment, if multiple host machines meet the set condition, the multiple host machines that meet the set condition are scored, and the host machine with the highest score is taken as the first target host machine.

[0036] S130, deploying the target master node to the first target host machine.

[0037] Specifically, after the first target host machine is determined, the target master node is deployed to the first target host machine.

[0038] Optionally, the process of deploying the target master node to the first target host machine can be: scheduling the target master node; and deploying the scheduled target master node to the first target host machine.

[0039] In this embodiment, the master nodes are scheduled by the K8s default scheduler and the custom scheduler extended based on the K8s default scheduler.

[0040] Optionally, before scheduling the target master node, the method further includes: determining whether other master nodes in the Redis cluster are in a scheduling state; and if the other master nodes are in the scheduling state, waiting until the other master nodes are scheduled to complete before scheduling the target master node.

[0041] In this embodiment, after the target master node and the first target host machine are determined, it is first determined whether there is a master node in the Redis cluster that is in a scheduling state, and if there is, the target master node is continued to be scheduled after the other master nodes are scheduled to complete, so that the scheduled target master node is deployed to the first target host machine.

[0042] Specifically, after the target master node is deployed to the first target host machine, the method further includes: updating the number of deployed master nodes of the first target host machine.

[0043] In this embodiment, after the target master node is deployed to the first target host machine, the number of deployed master nodes of the first target host machine is updated, which is beneficial to subsequently accurately deploying the nodes that have not been deployed to the host machine and preventing the deployment of the Redis cluster from being affected by data differences.

[0044] Optionally, the method further includes: deploying a slave node corresponding to the target master node to a second target host machine.

[0045] The second target host is different from the first target host. In this embodiment, the Pod anti-affinity configuration is used to schedule the slave nodes by using the default scheduler of K8s.

[0046] In this embodiment, after the target master node is deployed on the first target host, the operation of obtaining an undeployed master node in the Redis cluster as a target master node is returned to be executed until all nodes in the Redis cluster are deployed on the host.

[0047] The technical scheme of this embodiment is to obtain an undeployed master node in the Redis cluster as a target master node, and determine a host satisfying a set condition as a first target host. The set condition is that the proportion of the number of deployed master nodes in the Redis cluster does not exceed a set threshold. The target master node is deployed on the first target host. The Redis cluster deployment method provided in the embodiment can ensure the high availability of the Redis cluster, improve the resource utilization of the host, reduce the hardware cost, and ensure the availability of the Redis cluster even in the case of a small number of hosts and a large Redis cluster shard size.

[0048] Embodiment Two

[0049] Figure 3 is a structural schematic diagram of a Redis cluster high-availability deployment device provided in Embodiment Two of the application. The Redis cluster includes a plurality of shards, and each shard includes a master node and a slave node. As shown in the figure, the device includes: Figure 3

[0050] The target master node obtaining module 310 is configured to obtain an undeployed master node in the Redis cluster as a target master node.

[0051] The first target host determining module 320 is configured to determine a host satisfying a set condition as a first target host. The set condition is that the proportion of the number of deployed master nodes in the Redis cluster does not exceed a set threshold.

[0052] The node deployment module 330 is configured to deploy the target master node on the first target host.

[0053] Optionally, the target master node obtaining module 310 is further configured to:

[0054] obtain a preconfigured regular expression.

[0055] ​According to the regular expression, a master node is determined from the undeployed nodes.

[0056] Optionally, the first target host determining module 320 is further configured to:

[0057] obtain a total number of master nodes included in the Redis cluster;

[0058] for each host, obtain a number of deployed master nodes of the host;

[0059] determine a number ratio according to the number of deployed master nodes and the total number of master nodes;

[0060] if the number ratio does not exceed a set threshold, the host satisfies a set condition.

[0061] Optionally, the apparatus further comprises an updating module configured to:

[0062] update the number of deployed master nodes of the first target host.

[0063] Optionally, the node deploying module 330 is further configured to:

[0064] schedule the target master node;

[0065] deploy the scheduled target master node to the first target host.

[0066] Optionally, the apparatus further comprises a judging module configured to:

[0067] judge whether other master nodes in the Redis cluster are in a scheduling state;

[0068] if the other master nodes are in the scheduling state, schedule the target master node after the other master nodes are scheduled.

[0069] Optionally, the apparatus further comprises a slave node scheduling module configured to: deploy a slave node corresponding to the target master node to a second target host; wherein the second target host is different from the first target host.

[0070] The apparatus can perform the method provided by all the foregoing embodiments of the application, and has the corresponding function modules and beneficial effects of performing the foregoing method. Technical details not described in the embodiment can be referred to the method provided by all the foregoing embodiments of the application.

[0071] Embodiment Three

[0072] Figure 4A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0073] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0074] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the deployment methods for high availability of Redis clusters.

[0076] In some embodiments, the Redis cluster high availability deployment method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the Redis cluster high availability deployment method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the Redis cluster high availability deployment method by other means, e.g., with the aid of firmware.

[0077] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0078] Computer programs implementing methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0079] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0080] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0081] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0082] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0083] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0084] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A highly available deployment method for Redis clusters, suitable for situations with a small number of host machines and a large Redis cluster shard size, characterized in that... The Redis cluster comprises multiple shards, and each shard includes a master node and slave nodes; the method includes: Obtain an undeployed master node in the Redis cluster and use it as the target master node; A host machine that meets the set conditions is identified as the first target host machine; wherein, the set conditions are that the proportion of master nodes in the deployed Redis cluster does not exceed a set threshold. The target master node is deployed on the first target host machine; Among them, the host machines that meet the set conditions are identified as the first target host machines, including: Get the total number of master nodes in the Redis cluster; For each host machine, obtain the number of master nodes deployed on that host machine; The quantity ratio is determined based on the number of deployed master nodes and the total number of master nodes; If the quantity ratio does not exceed the set threshold, then the host machine meets the set conditions; The step of determining the quantity ratio based on the number of deployed master nodes and the total number of master nodes includes: Divide the number of deployed master nodes by the total number of master nodes in the Redis cluster to obtain the proportion of deployed master nodes.

2. The method according to claim 1, characterized in that, Obtaining an undeployed master node in the Redis cluster includes: Retrieve pre-configured regular expressions; A master node is determined from the undeployed nodes based on the regular expression.

3. The method according to claim 1, characterized in that, After deploying the target master node on the first target host machine, the method further includes: Update the number of deployed master nodes on the first target host machine.

4. The method according to claim 1, characterized in that, Deploying the target master node on the first target host machine includes: Schedule the target master node; The scheduled target master node is deployed on the first target host machine.

5. The method according to claim 4, characterized in that, Before scheduling the target master node, the following steps are also included: Determine whether other master nodes in the Redis cluster are in a scheduling state; If other master nodes are in a scheduling state, the target master node will be scheduled after the other master nodes have finished scheduling.

6. The method according to claim 1, characterized in that, Also includes: The slave node corresponding to the target master node is deployed on the second target host machine; wherein the second target host machine is different from the first target host machine.

7. A highly available deployment device for a Redis cluster, suitable for situations where the number of host machines is small and the Redis cluster shard size is large, characterized in that, The Redis cluster comprises multiple shards, and each shard includes a master node and slave nodes; the device includes: The target master node acquisition module is used to acquire an undeployed master node in the Redis cluster as the target master node. The first target host determination module is used to determine a host that meets the set conditions as the first target host; wherein, the set conditions are that the proportion of the number of master nodes deployed in the Redis cluster does not exceed the set threshold. A node deployment module is used to deploy the target master node on the first target host machine; The first target host determination module is further configured to: Get the total number of master nodes in the Redis cluster; For each host machine, obtain the number of master nodes deployed on that host machine; The quantity ratio is determined based on the number of deployed master nodes and the total number of master nodes; If the quantity ratio does not exceed the set threshold, then the host machine meets the set conditions; The step of determining the quantity ratio based on the number of deployed master nodes and the total number of master nodes includes: Divide the number of deployed master nodes by the total number of master nodes in the Redis cluster to obtain the proportion of deployed master nodes.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the Redis cluster high availability deployment method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the deployment method for high availability of the Redis cluster as described in any one of claims 1-6.

Images