Fault detection method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a fault detection method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment, aiming at each Redis node, determining a node type of the Redis node identified by each Redis node according to each node information set, and then determining a fault detection result of the Redis cluster according to the number of first nodes of which the Redis nodes are identified as the same node type.

Description

Technical Field

[0001] The present application relates to the technical field of fault detection, and in particular to a fault detection method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In modern industrial and information systems, the control module, as the core component of the system, is responsible for monitoring, regulating and coordinating the operation of various subsystems. Its stability and reliability are directly related to the performance and safety of the entire system. However, due to factors such as long-term operation, complex environment and equipment aging, the control module may encounter various faults, which may lead to serious consequences such as system performance degradation and data loss. Therefore, it is particularly important to detect and handle control module faults.

[0003] Currently, control module fault detection is usually performed on the controller. However, in addition to the controller, the control module also has a remote dictionary server (Redis) cluster involved in the operation. Therefore, how to detect Redis cluster faults has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] Based on this, it is necessary to provide a fault detection method, device, computer equipment, storage medium and program product that can detect Redis cluster faults in response to the above technical problems.

[0005] In a first aspect, the present application provides a fault detection method. The method comprises:

[0006] Obtaining a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node;

[0007] For each of the Redis nodes, determine the node type of the Redis node recognized by each of the Redis nodes according to each of the node information sets; the node type includes at least one of being in the Redis cluster, not being in the Redis cluster, a primary node, and a standby node;

[0008] A fault detection result of the Redis cluster is determined according to the number of first nodes of the same node type that are identified as the Redis nodes.

[0009] In one embodiment, determining the fault detection result of the Redis cluster according to the number of first nodes of the same node type identified as the Redis node includes:

[0010] If the number of the first nodes is greater than or equal to a first preset threshold, the same node type is used as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; the first preset threshold is determined according to the total number of Redis nodes in the Redis cluster;

[0011] If the number of first nodes of the same node type is equal to half of the total number, it is determined that the fault detection result is a Redis cluster fault.

[0012] In one embodiment, the method further comprises:

[0013] When it is determined that the fault detection result is that the Redis cluster is normal, the number of second nodes of the Redis nodes of the target type being the primary node deployed on a single device is obtained;

[0014] If the number of the second nodes is greater than the second preset threshold, the Redis node whose target type is the master node on the single device is migrated.

[0015] In one embodiment, the migrating of the Redis node whose target type is a master node on the single device includes:

[0016] Determine the node to be migrated from the Redis nodes of the target type that are master nodes;

[0017] Determine the number of third nodes of the master node on the target device where the standby node corresponding to the node to be migrated is located;

[0018] If the third node number is less than the second preset threshold, the node to be migrated is migrated to the target device.

[0019] In one embodiment, the method further comprises:

[0020] When it is determined that the fault detection result is a fault of the Redis cluster, obtaining a storage location and a storage method of the first resource data of the first control environment from a resource data table;

[0021] According to the storage location and the storage method, the first resource data is acquired and stored in a second control environment; the second control environment is a backup control environment for the first control environment.

[0022] In one embodiment, the method further comprises:

[0023] When it is determined that the fault detection result is a fault of the Redis cluster, obtaining first business data of the first business process in the first control environment from the state data table;

[0024] The first business process is restored in the second control environment according to the first business data.

[0025] In a second aspect, the present application also provides a fault detection device. The device comprises:

[0026] A first acquisition module is used to acquire a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node;

[0027] A first determination module is used to determine, for each Redis node, a node type recognized by each Redis node according to each node information set; the node type includes at least one of being in the Redis cluster, not in the Redis cluster, a primary node, and a standby node;

[0028] The second determination module is used to determine the fault detection result of the Redis cluster according to the number of first nodes of the same node type that are identified as the Redis nodes.

[0029] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0031] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0032] The above-mentioned fault detection method, apparatus, computer equipment, storage medium and program product obtain the node information set of each Redis node in the remote dictionary service Redis cluster in the first control environment, and for each Redis node, determine the node type recognized by each Redis node according to each node information set, and then determine the fault detection result of the Redis cluster according to the number of first nodes recognized as the same node type of the Redis node, thereby realizing the detection of faults in the Redis cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is an internal structure diagram of a computer device provided in an embodiment of the present application;

[0034] FIG2 is a schematic diagram of a flow chart of a fault detection method provided in an embodiment of the present application;

[0035] FIG3 is a schematic diagram of the structure of a control module provided in an embodiment of the present application;

[0036] FIG4 is a flow chart of a method for determining a fault detection result provided in an embodiment of the present application;

[0037] FIG5 is a schematic diagram of a process flow of a master node migration method provided in an embodiment of the present application;

[0038] FIG6 is a schematic diagram of a flow chart of another master node migration method provided in an embodiment of the present application;

[0039] 7 is a flow chart of a Redis cluster fault detection and dynamic rebalancing method provided in an embodiment of the present application;

[0040] FIG8 is a schematic diagram of a flow chart of a resource data migration method provided in an embodiment of the present application;

[0041] 9 is a schematic diagram of a process flow of a business process recovery method provided in an embodiment of the present application;

[0042] 10 is a schematic diagram of a process flow of another business process recovery method provided in an embodiment of the present application;

[0043] FIG. 11 is a structural block diagram of a fault detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In modern industrial and information systems, the control module, as the core component of the system, is responsible for monitoring, regulating and coordinating the operation of various subsystems. Its stability and reliability are directly related to the performance and safety of the entire system. However, due to factors such as long-term operation, complex environment and equipment aging, the control module may encounter various faults, which may lead to serious consequences such as system performance degradation and data loss. Therefore, it is particularly important to detect and handle control module faults.

[0046] Currently, control module fault detection is usually performed on the controller. However, in addition to the controller, the control module also has a remote dictionary server (Redis) cluster involved in the operation. Therefore, how to detect Redis cluster faults has become a technical problem that needs to be solved urgently in this field.

[0047] The fault detection method provided in the embodiment of the present application can be applied in the application environment as shown in Figure 1. Figure 1 is an internal structure diagram of a computer device provided in the embodiment of the present application. The computer device can be a server, and its internal structure diagram can be as shown in Figure 1. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fault detection method is implemented.

[0048] Those skilled in the art will understand that the structure shown in FIG. 1 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0049] In one embodiment, as shown in FIG. 2 , FIG. 2 is a flow chart of a fault detection method provided in an embodiment of the present application, which can be applied to the computer device in FIG. 1 , and the method includes the following steps:

[0050] S201, obtaining a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment.

[0051] The node information set includes the node information of each Redis node.

[0052] Exemplarily, refer to Figure 3, which is a schematic diagram of the structure of a control module provided in an embodiment of the present application. As shown in Figure 3, the control module may include two control environments, which are mirrored and deployed. The first control environment is the main control environment, and the second control environment is the backup control environment of the first control environment. When the first control environment fails, it can be switched to the second control environment to continue to execute the corresponding business process. Redis in the control module usually undertakes functions such as data caching and distributed locks, and is an important part of the control module.

[0053] In one embodiment, the Redis in the control environment can be deployed in a cluster manner. As shown in FIG3 , a control environment includes at least one Redis cluster, and a Redis cluster includes multiple Redis nodes.

[0054] In the embodiment of the present application, each Redis node in the Redis cluster in the first control environment stores a node information set, and the node information set includes the node information of each Redis node in the Redis cluster. That is to say, assuming that there are N Redis nodes in the Redis cluster in the first control environment, each Redis node stores a node information set, and the node information set includes the node information of N Redis nodes.

[0055] In the embodiment of the present application, the node information of the Redis node may include, for example, the device identifier (device ip) of the device where the Redis node is deployed, the port number (port) of the Redis node, the role (role), and the state (state). If the role of the Redis node is the master node, the node information of the Redis node also includes the identifier (slaveIp) and port number (slavePort) of the corresponding slave node.

[0056] The role of a Redis node may include, for example, a master node and a slave node; the state may include, for example, being in a Redis cluster (connected) and not being in a Redis cluster (failed).

[0057] Optionally, the above node information set can be saved in a file named "nodes-node port number.conf", and the node information of each Redis node can be determined by parsing the data stored in the "nodes-node port number.conf" file, and the node information of each Redis node is stored in the form of map data. The form of map data can be, for example: map <device ip,list <port:role:state:slaveip:slaveport>>.

[0058] S202: for each Redis node, determine the node type recognized by each Redis node according to each node information set.

[0059] The node type includes at least one of being in a Redis cluster, not being in a Redis cluster, a master node, and a standby node.

[0060] In the embodiment of the present application, the node type cannot include both the master node and the standby node, and cannot include both being in the Redis cluster and not being in the Redis cluster at the same time.

[0061] In one embodiment, since the role or status of the same Redis node may be different from the perspective of different Redis nodes, the node information for the same Redis node stored in different node information sets may also be different. In other words, for each Redis node, the identified node type of the Redis node may also be different according to different node information sets.

[0062] Exemplarily, assuming that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, and node 6, for node 1, the node type of node 1 recognized by node 1 can be determined from the node information set of node 1, the node type of node 1 recognized by node 2 can be determined from the node information set of node 2, the node type of node 1 recognized by node 3 can be determined from the node information set of node 3, the node type of node 1 recognized by node 4 can be determined from the node information set of node 4, the node type of node 1 recognized by node 5 can be determined from the node information set of node 5, and the node type of node 1 recognized by node 6 can be determined from the node information set of node 6.

[0063] S203: Determine a fault detection result of the Redis cluster according to the number of first nodes that are identified as the same node type as the Redis nodes.

[0064] In an embodiment of the present application, a first preset threshold may be determined according to the total number of Redis nodes in the Redis cluster, and then a fault detection result of the Redis cluster may be determined according to the first preset threshold and the first number of nodes.

[0065] Optionally, half of the total number of Redis nodes in the Redis cluster may be determined as the first preset threshold.

[0066] Alternatively, the first preset threshold may be determined as half of the total number of Redis nodes in the Redis cluster plus 1. That is, the first preset threshold may be determined according to the following formula:

[0067] z = m / 2

[0068] n = z + 1

[0069] Wherein, m is the total number of Redis nodes in the Redis cluster, and n is the first preset threshold.

[0070] In one embodiment, when n is the first preset threshold, if the first number of nodes is greater than or equal to the first preset threshold, the same node type is determined as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal. If the first number of nodes cannot be greater than or equal to the first preset threshold, that is, when the target type of the Redis node cannot be determined, the fault detection result is determined to be a Redis cluster failure.

[0071] Exemplarily, assume that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, and node 6, the first preset threshold is 4, and the node type of node 1 identified by node 1 is the primary node, the node type of node 1 identified by node 2 is the primary node, the node type of node 1 identified by node 3 is the primary node, the node type of node 1 identified by node 4 is the standby node, the node type of node 1 identified by node 5 is the standby node, and the node type of node 1 identified by node 6 is the primary node, then the first number of nodes identified as the primary node by node 1 is 4, which is equal to the first preset threshold, and the primary node is determined as the target type of node 1.

[0072] Alternatively, assuming that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, and node 6, the first preset threshold is 4, and the node type of node 1 identified by node 1 is a primary node, the node type of node 1 identified by node 2 is a primary node, the node type of node 1 identified by node 3 is a standby node, the node type of node 1 identified by node 4 is a standby node, the node type of node 1 identified by node 5 is a standby node, and the node type of node 1 identified by node 6 is a primary node, then the first number of nodes identified by node 1 as the primary node is 3, the first number of nodes identified as the standby node is also 3, and the first number of nodes cannot be greater than or equal to the first preset threshold, then the fault detection result is determined to be a Redis cluster failure.

[0073] In another embodiment, when z is the first preset threshold, if the first number of nodes is greater than the first preset threshold, the same node type is determined as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; if the first number of nodes is equal to the first preset threshold, the fault detection result is determined to be a Redis cluster fault.

[0074] Exemplarily, assume that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, and node 6, the first preset threshold is 4, and the node type of node 1 identified by node 1 is a master node, the node type of node 1 identified by node 2 is a master node, the node type of node 1 identified by node 3 is a standby node, the node type of node 1 identified by node 4 is a standby node, the node type of node 1 identified by node 5 is a master node, and the node type of node 1 identified by node 6 is a master node, then the first preset threshold is 3, and the first number of nodes identified as the master node is 4, which is greater than the first preset threshold, then the master node is determined as the target type of node 1.

[0075] Alternatively, assuming that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, and node 6, the first preset threshold is 4, and the node type of node 1 identified by node 1 is a primary node, the node type of node 1 identified by node 2 is a primary node, the node type of node 1 identified by node 3 is a standby node, the node type of node 1 identified by node 4 is a standby node, the node type of node 1 identified by node 5 is a standby node, and the node type of node 1 identified by node 6 is a primary node, then the first preset threshold is 3, the first number of nodes identified as the primary node is 3, the first number of nodes identified as the standby node is also 3, and the first number of nodes is equal to the first preset threshold, then the fault detection result is determined to be a Redis cluster failure.

[0076] In an embodiment of the present application, by obtaining the node information set of each Redis node in the remote dictionary service Redis cluster in the first control environment, for each Redis node, based on each node information set, the node type recognized by each Redis node is determined, and then based on the number of first nodes of the Redis node recognized as the same node type, the fault detection result of the Redis cluster is determined.

[0077] Referring to FIG. 4 , FIG. 4 is a flow chart of a method for determining a fault detection result provided by an embodiment of the present application. This embodiment relates to a possible implementation method of determining a fault detection result of a Redis cluster based on the number of first nodes identified as the same node type by the Redis nodes. Based on the above embodiment, the above S203 includes the following steps:

[0078] S401: If the number of first nodes is greater than or equal to a first preset threshold, the same node type is used as a target type of the Redis node, and a fault detection result is determined to be that the Redis cluster is normal.

[0079] The first preset threshold is determined according to the total number of Redis nodes in the Redis cluster.

[0080] In the embodiment of the present application, the first preset threshold can be determined according to the following formula:

[0081] z = m / 2

[0082] n = z + 1

[0083] Wherein, m is the total number of Redis nodes in the Redis cluster, and n is the first preset threshold.

[0084] In one embodiment, if the first number of nodes is greater than or equal to a first preset threshold, the same node type is determined as a target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal.

[0085] Exemplarily, assume that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, and node 6, the first preset threshold is 4, and the node type of node 1 identified by node 1 is the primary node, the node type of node 1 identified by node 2 is the primary node, the node type of node 1 identified by node 3 is the primary node, the node type of node 1 identified by node 4 is the standby node, the node type of node 1 identified by node 5 is the standby node, and the node type of node 1 identified by node 6 is the primary node, then the first number of nodes identified as the primary node by node 1 is 4, which is equal to the first preset threshold, and the primary node is determined as the target type of node 1.

[0086] It should be noted that only when all Redis nodes in the Redis cluster can determine the target type, can the fault detection result of the Redis cluster be determined to be normal. As long as there is a Redis node in the Redis cluster that cannot determine the target type, the fault detection result of the Redis cluster is a Redis cluster failure.

[0087] S402: If the number of first nodes of the same node type is equal to half of the total number, it is determined that the fault detection result is a Redis cluster fault.

[0088] In the embodiment of the present application, if the number of first nodes of the same node type is equal to half of the total number, it is determined that the fault detection result is a Redis cluster failure.

[0089] Exemplarily, assume that there are 6 Redis nodes in the Redis cluster, namely: node 1, node 2, node 3, node 4, node 5, node 6, the first preset threshold is 4, and the node type of node 1 identified by node 1 is a primary node, the node type of node 1 identified by node 2 is a primary node, the node type of node 1 identified by node 3 is a standby node, the node type of node 1 identified by node 4 is a standby node, the node type of node 1 identified by node 5 is a standby node, and the node type of node 1 identified by node 6 is a primary node, then the first number of nodes identified as primary nodes by node 1 is 3, the first number of nodes identified as standby nodes is also 3, and the first number of nodes is equal to half of the total number, then the fault detection result is determined to be a Redis cluster failure.

[0090] In an embodiment of the present application, a first preset threshold is determined according to the total number of Redis nodes in the Redis cluster. If the first number of nodes is greater than or equal to the first preset threshold, the same node type is used as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; if the first number of nodes of the same node type is equal to half of the total number, the fault detection result is determined to be a Redis cluster failure, thereby realizing the detection of Redis cluster failures, and then being able to switch the control environment in time when a Redis cluster failure occurs, thereby improving the stability and reliability of the controller service.

[0091] Referring to Figure 5, Figure 5 is a flow chart of a primary node migration method provided by an embodiment of the present application. Based on the above embodiment, the method further includes the following steps:

[0092] S501, when it is determined that the fault detection result is that the Redis cluster is normal, obtain the number of second nodes of the Redis nodes whose target type is the master node deployed on a single device.

[0093] Optionally, the Redis cluster usually includes M Redis nodes whose target type is a master node, and M Redis nodes whose target type is a standby node set corresponding to each master node. Wherein, M is an odd number. The 2M Redis nodes in the Redis cluster can be deployed on several devices.

[0094] Under perfect deployment requirements, 2M Redis nodes should be deployed on 2M independent devices. In the case of insufficient deployment resources, multiple Redis nodes may be deployed on one device. If the master node deployment is too centralized, that is, too many master nodes are deployed on the same device, if the device fails, the Redis cluster will fail. Therefore, the centralized master nodes need to be migrated.

[0095] In the embodiment of the present application, when it is determined that the fault detection result is that the Redis cluster is normal, the target type of each Redis node in the Redis cluster can be determined, and then the number of second nodes of the Redis nodes whose target type is the master node deployed on a single device can be determined. The single device can be any device that deploys the Redis node.

[0096] S502: If the number of second nodes is greater than the second preset threshold, migrate the Redis node whose target type is the master node on a single device.

[0097] Exemplarily, assuming that the number of Redis nodes whose target type is primary nodes in the Redis cluster is z, and the total number of devices on which Redis nodes are deployed is x, when z < x, the second preset threshold can be set to 2, that is, when the second number of nodes is greater than 2, the Redis nodes whose target type is primary nodes on a single device are migrated.

[0098] Alternatively, when z ≧ x, the second preset threshold can be set to z / x + ((z % x) != 0 ? 1 : 0), that is, when the second number of nodes is greater than z / x + ((z % x) != 0 ? 1 : 0), the Redis node with the target type of the master node on a single device is migrated.

[0099] Among them, " / " is the division symbol, that is, only the integer part of the quotient of z divided by x is retained, and the decimal is discarded and not rounded. "(z % x) != 0 ? 1 : 0" is a ternary expression, which means that when the remainder of z divided by x is not equal to 0 (that is, z cannot be divided by x), the result of the expression is 1, otherwise the result of the expression is 0.

[0100] In a possible implementation, when determining that a Redis node whose target type is a primary node on a single device needs to be migrated, all Redis nodes whose target type is a primary node on a single device can be traversed, and first, one of the Redis nodes whose target type is a primary node on a single device is selected as the node to be migrated, and then the number of third nodes of the primary node on the target device where the backup node corresponding to the node to be migrated is located is determined, and the number of third nodes is compared with the second preset threshold. If the number of third nodes is less than the second preset threshold, the node to be migrated is changed into a backup node, and the backup node corresponding to the node to be migrated is changed into the primary node to achieve the migration of the primary node. If the number of third nodes is not less than the second preset threshold, another Redis node among the Redis nodes whose target type is a primary node on a single device is used as the node to be migrated, and then the comparison process of the number of third nodes and the second preset threshold is repeated until the primary node that can be migrated is found, and the migration of the primary node is completed.

[0101] In an embodiment of the present application, when it is determined that the fault detection result is that the Redis cluster is normal, the second node number of Redis nodes whose target type is the master node deployed on a single device is obtained. If the second node number is greater than the second preset threshold, the Redis nodes whose target type is the master node on the single device are migrated. Therefore, when the master nodes on a single device are overly concentrated, the master nodes on the single device can be migrated, thereby avoiding the failure of the Redis cluster due to the failure of a single device, thereby improving the safety and reliability of the control module operation.

[0102] Referring to Figure 6, Figure 6 is a flow chart of another master node migration method provided by an embodiment of the present application. This implementation relates to a possible implementation method of how to migrate a Redis node whose target type is a master node on a single device. Based on the above embodiment, the above S502 includes the following steps:

[0103] S601, determining a node to be migrated from Redis nodes whose target type is a master node.

[0104] In one embodiment, when it is determined that a Redis node whose target type is a master node on a single device needs to be migrated, one of the Redis nodes whose target type is a master node on the single device may be selected as the node to be migrated.

[0105] S602, determine the third node number of the master node on the target device where the standby node corresponding to the node to be migrated is located.

[0106] In an embodiment of the present application, the node information of the standby node corresponding to the node to be migrated can be determined according to the node information of the node to be migrated, and then the third node number of the master node on the target device where the standby node is located can be determined according to the node information of the standby node.

[0107] S603: If the number of the third nodes is less than the second preset threshold, migrate the node to be migrated to the target device.

[0108] In one embodiment, the number of the third nodes can be compared with the second preset threshold. If the number of the third nodes is less than the second preset threshold, the node to be migrated is changed to a standby node, and the standby node corresponding to the node to be migrated is changed to a master node to realize the migration of the master node.

[0109] In another embodiment, if the number of the third nodes is not less than the second preset threshold, another Redis node among the Redis nodes whose target type is the master node on a single device is used as the node to be migrated, and then the comparison process of the above-mentioned number of the third nodes and the second preset threshold is repeated until the master node that can be migrated is found and the migration of the master node is completed.

[0110] In an embodiment of the present application, the node to be migrated is determined from the Redis node whose target type is the master node, and the third node number of the master node on the target device where the standby node corresponding to the node to be migrated is located is determined. If the third node number is less than the second preset threshold, the node to be migrated is migrated to the target device. In this way, when the master node deployment on a single device is overly concentrated, the master node on a single device can be migrated, thereby avoiding the failure of the Redis cluster due to the failure of a single device, thereby improving the safety and reliability of the control module operation.

[0111] In order to introduce the embodiment of the present application more clearly, an exemplary description is given here in conjunction with Figure 7. Figure 7 is a flow chart of a Redis cluster fault detection and dynamic rebalancing method provided by an embodiment of the present application. As shown in Figure 7, the node information of each Redis node in the Redis cluster can be obtained and stored in the form of map data, and then the node information of each Redis node in the Redis cluster is determined. The information set of each node is determined based on the information in each node information set. Whether the Redis cluster fails, when the Redis cluster fails, the fault handling process is started, and the corresponding business process is continued to be executed in the second control environment. When the Redis cluster is normal, that is, when there is no failure, it is determined whether the master node deployed on each device is reasonable. When it is determined that the master node deployed on a single device is unreasonable, it is determined to migrate the Redis node whose target type is the master node on the single device. Then traverse the map data of the Redis node whose target type is the master node on the single device, find the node to be migrated, and determine the standby node corresponding to the node to be migrated according to the list in the map data of the node to be migrated. Determine whether the target device where the standby node is located meets the migration condition, that is, compare the number of the third node with the second preset threshold value. If the number of the third node is less than the second preset threshold value, the target device where the standby node is located meets the migration condition, and the node to be migrated is changed into a standby node, and the standby node corresponding to the node to be migrated is changed into a master node to realize the migration of the master node. If the target device where the standby node is located does not meet the migration condition, repeat the above steps until a master node that can be migrated is found, and the migration of the master node is completed.

[0112] Referring to Figure 8, Figure 8 is a flow chart of a resource data migration method provided in an embodiment of the present application. Based on the above embodiment, the method further includes the following steps:

[0113] S801: When it is determined that the fault detection result is a Redis cluster fault, a storage location and a storage method of first resource data of a first control environment are obtained from a resource data table.

[0114] In the embodiment of the present application, the resource type, resource identifier, storage location, and storage method of the first resource data temporarily stored in the Redis cluster in the first control environment may be saved in a resource data table.

[0115] Exemplarily, the resource data table may be as shown in Table 1 below, where the first column in Table 1 is used to store the resource type of the first resource data, the second column is used to store the resource identifier of the first resource data, the third column is used to store the storage location of the first resource data, and the fourth column is used to store the storage method of the first resource data. The storage location of the first resource data may be a table item of a data table storing the first resource data, and the storage method may be a combination method of the fields in the data table storing the first resource data.

[0116]

[0117] Table 1

[0118] In one embodiment, when it is determined that the fault detection result is a Redis cluster fault, it is necessary to switch to the second control environment to continue executing the corresponding business process. At this time, the storage location and storage method of the first resource data can be obtained from the resource data table.

[0119] S802: Acquire first resource data and store the first resource data in a second control environment according to a storage location and a storage method.

[0120] The second control environment is a backup control environment for the first control environment.

[0121] In one embodiment, the storage location of the first resource data may be traversed, and the resource data stored in the storage location may be combined according to the storage mode to restore the first resource data.

[0122] In addition, the first resource data may also be stored in the second control environment according to a storage location and a storage method.

[0123] In an embodiment of the present application, when it is determined that the fault detection result is a Redis cluster failure, the storage location and storage method of the first resource data of the first control environment are obtained from the resource data table, and according to the storage location and storage method, the first resource data is obtained and stored in the second control environment, so that when a failure occurs in the first control environment and it switches to the second control environment, the first resource data can be restored in the second control environment, thereby ensuring the continuity of the business, so that the control module can quickly resume operations after the control environment is switched.

[0124] Referring to Figure 9, Figure 9 is a flow chart of a method for recovering a business process provided by an embodiment of the present application. Based on the above embodiment, the method further includes the following steps:

[0125] S901: When it is determined that the fault detection result is a Redis cluster fault, first business data of a first business process in a first control environment is obtained from a state data table.

[0126] In the embodiment of the present application, the business process of the control module can be divided into five steps, namely: parameter verification, configuration construction, configuration distribution, linkage processing and data storage. Among them, parameter verification is to check the existence of parameters in the input parameters, the correctness of the format, and the matching of the associated attributes. The input is the input parameter, and the output is the error information checked. Configuration construction is the process of constructing the configuration type, configuration parameters, and configuration format of the business according to the input parameters and business type requirements. The input is the business type and key parameters, and the output is the configuration format pointing to the device. Configuration distribution pushes the business configuration to the device pointed to by the configuration constructed in the previous step and records the push results. The input is the business configuration, and the output is the configuration push result. Linkage processing is when the business requires joint processing of multiple services, calling the linkage processing service to notify it to do associated business processing. The input is the business type and business key parameters, and the output is the processing result of the associated service. Data storage is the process of saving business-related data. The input is the data generated by the business, and the output is the storage result. The above five steps must comply with the idempotence principle and maintain the execution order for each business.

[0127] In one embodiment, when the first business process reaches the first control environment, the first business data of the first business process may be stored in a state data table.

[0128] Exemplarily, the status data table can be shown in Table 2 below, where the first column in Table 2 is the business type, which is used to save the type of business; the second column is the input parameters, which are assembled according to colons and saved; the third column is the business step, which records which step of the above five steps the current business is in; the fourth column is the input, where multiple parameters are separated by semicolons, and the input of the current step is recorded; the fifth column is the status, which records the execution status of the current step, which is divided into three types: processing, success, and failure; the sixth column records the control environment of the processing, which is divided into the first control environment and the second control environment.

[0129]

[0130] Table 2

[0131] S902: Restore the first business process in the second control environment according to the first business data.

[0132] Exemplarily, in one embodiment, when it is determined that the fault detection result is a Redis cluster failure, it is necessary to switch to the second control environment to continue executing the corresponding business process. At this time, the sixth column in the status data table can be traversed, and the first business process in the first control environment can be filtered out according to the control environment stored in the sixth column, and then the execution status of the first business process is obtained. If the execution status of the first business process is successful, the data is deleted from the status data table; if the execution status of the first business process is a failure, the data is deleted from the status data table after the rollback process; if the execution status of the first business process is in progress, the first business process can be restored in the second control environment according to the first business data in the status data table.

[0133] Exemplarily, refer to Figure 10, which is a flow chart of another business process recovery method provided by an embodiment of the present application. As shown in Figure 10, when it is determined that the fault detection result is a Redis cluster fault, it is necessary to switch to the second control environment to continue to execute the corresponding business process. At this time, the first business process can be obtained from the database. When it is determined that the first business process exists, the sixth column in the status data table is traversed, and the first business process in the first control environment is screened out according to the control environment stored in the sixth column, and then the execution status of the first business process is obtained. If the execution status of the first business process is successful, the data is deleted from the status data table; if the execution status of the first business process is a failure, the data is deleted from the status data table after the rollback process; if the execution status of the first business process is in progress, the first business process can be restored in the second control environment according to the first business data in the status data table.

[0134] In one embodiment, the data stored in the database of the first control environment and the database of the second control environment can also be reconciled regularly: the status data tables in the first control environment and the second control environment are traversed, the number of business data in each status data table is counted, and the number of business data in the first control environment and the second control environment is compared. If the numbers are the same, the data is considered to be consistent. If the numbers are different, it is considered that there may be differences in the configuration. If there are inconsistent numbers for three consecutive reconciliations, a detailed reconciliation is triggered, and the business data in the status data tables in the first control environment and the second control environment are compared one by one. If the business data is inconsistent, an alarm message is generated to remind relevant managers that the business data in the first control environment and the second control environment are inconsistent, so that relevant managers can take timely processing measures.

[0135] In an embodiment of the present application, when it is determined that the fault detection result is a Redis cluster failure, the first business data of the first business process in the first control environment is obtained from the status data table, and the first business process is restored in the second control environment based on the first business data. Therefore, when a failure occurs in the first control environment and it is switched to the second control environment, the first business process can be restored in the second control environment, thereby ensuring the continuity of the business, so that the control module can quickly resume operations after the control environment is switched.

[0136] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present application also provides a fault detection device for implementing the fault detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault detection device embodiments provided below can refer to the limitations of the fault detection method above, and will not be repeated here.

[0138] In one embodiment, as shown in FIG. 11 , FIG. 11 is a structural block diagram of a fault detection device provided in an embodiment of the present application, and the device 1100 includes:

[0139] The first acquisition module 1101 is used to acquire a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node.

[0140] The first determination module 1102 is used to determine, for each Redis node, the node type recognized by each Redis node according to each node information set; the node type includes at least one of being in a Redis cluster, not being in a Redis cluster, a primary node, and a backup node.

[0141] The second determination module 1103 is used to determine the fault detection result of the Redis cluster according to the number of first nodes that are identified as the same node type as the Redis nodes.

[0142] In one embodiment, the second determining module 1103 includes:

[0143] The first determination unit is used to use the same node type as the target type of the Redis node if the number of first nodes is greater than or equal to a first preset threshold, and determine that the fault detection result is that the Redis cluster is normal; the first preset threshold is determined according to the total number of Redis nodes in the Redis cluster.

[0144] The second determining unit is used to determine that the fault detection result is a Redis cluster fault if the number of first nodes of the same node type is equal to half of the total number.

[0145] In one embodiment, the apparatus 1100 further includes:

[0146] The second acquisition module is used to acquire the number of second nodes of Redis nodes whose target type is a master node deployed on a single device when it is determined that the fault detection result is that the Redis cluster is normal.

[0147] The migration module is used to migrate the Redis node whose target type is the master node on a single device if the number of the second nodes is greater than the second preset threshold.

[0148] In one embodiment, the migration module includes:

[0149] The third determining unit is used to determine the node to be migrated from the Redis nodes whose target type is the master node.

[0150] The fourth determining unit is used to determine the third node number of the master node on the target device where the standby node corresponding to the node to be migrated is located.

[0151] The migration unit is configured to migrate the node to be migrated to the target device if the number of the third nodes is less than the second preset threshold.

[0152] In one embodiment, the apparatus 1100 further includes:

[0153] The third acquisition module is used to acquire the storage location and storage method of the first resource data of the first control environment from the resource data table when it is determined that the fault detection result is a Redis cluster fault.

[0154] The storage module is used to obtain the first resource data and store the first resource data in the second control environment according to the storage location and storage method; the second control environment is a backup control environment for the first control environment.

[0155] In one embodiment, the apparatus 1100 further includes:

[0156] The fourth acquisition module is used to acquire the first business data of the first business process in the first control environment from the state data table when it is determined that the fault detection result is a Redis cluster fault.

[0157] The recovery module is used to recover the first business process in the second control environment according to the first business data.

[0158] Each module in the above fault detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0159] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0160] Obtaining a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node;

[0161] For each Redis node, determine the node type of the Redis node recognized by each Redis node according to each node information set; the node type includes at least one of being in the Redis cluster, not being in the Redis cluster, a master node, and a standby node;

[0162] A fault detection result of the Redis cluster is determined according to the number of first nodes of the Redis nodes that are identified as the same node type.

[0163] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0164] If the number of the first nodes is greater than or equal to the first preset threshold, the same node type is used as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; the first preset threshold is determined according to the total number of Redis nodes in the Redis cluster;

[0165] If the number of first nodes of the same node type is equal to half of the total number, the fault detection result is determined to be a Redis cluster fault.

[0166] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0167] When it is determined that the fault detection result shows that the Redis cluster is normal, the number of second nodes of the Redis nodes whose target type is the primary node and which are deployed on a single device is obtained;

[0168] If the number of the second nodes is greater than the second preset threshold, the Redis node whose target type is the master node on the single device is migrated.

[0169] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0170] Determine the node to be migrated from the Redis nodes whose target type is the master node;

[0171] Determine the number of third nodes of the master node on the target device where the standby node corresponding to the node to be migrated is located;

[0172] If the third number of nodes is less than the second preset threshold, the node to be migrated is migrated to the target device.

[0173] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0174] When it is determined that the fault detection result is a Redis cluster fault, obtaining a storage location and a storage method of first resource data of the first control environment from a resource data table;

[0175] According to the storage location and storage method, the first resource data is acquired and stored in the second control environment; the second control environment is a backup control environment for the first control environment.

[0176] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0177] When it is determined that the fault detection result is a Redis cluster fault, obtaining first business data of a first business process in a first control environment from a state data table;

[0178] The first business process is restored in the second control environment according to the first business data.

[0179] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0180] Obtaining a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node;

[0181] For each Redis node, determine the node type of the Redis node recognized by each Redis node according to each node information set; the node type includes at least one of being in the Redis cluster, not being in the Redis cluster, a master node, and a standby node;

[0182] A fault detection result of the Redis cluster is determined according to the number of first nodes of the Redis nodes that are identified as the same node type.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] If the number of the first nodes is greater than or equal to the first preset threshold, the same node type is used as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; the first preset threshold is determined according to the total number of Redis nodes in the Redis cluster;

[0185] If the number of first nodes of the same node type is equal to half of the total number, the fault detection result is determined to be a Redis cluster fault.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0187] When it is determined that the fault detection result shows that the Redis cluster is normal, the number of second nodes of the Redis nodes whose target type is the primary node and which are deployed on a single device is obtained;

[0188] If the number of the second nodes is greater than the second preset threshold, the Redis node whose target type is the master node on the single device is migrated.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0190] Determine the node to be migrated from the Redis nodes whose target type is the master node;

[0191] Determine the number of third nodes of the master node on the target device where the standby node corresponding to the node to be migrated is located;

[0192] If the third number of nodes is less than the second preset threshold, the node to be migrated is migrated to the target device.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] When it is determined that the fault detection result is a Redis cluster fault, obtaining a storage location and a storage method of first resource data of the first control environment from a resource data table;

[0195] According to the storage location and storage method, the first resource data is acquired and stored in the second control environment; the second control environment is a backup control environment for the first control environment.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] When it is determined that the fault detection result is a Redis cluster fault, obtaining first business data of a first business process in a first control environment from a state data table;

[0198] The first business process is restored in the second control environment according to the first business data.

[0199] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0200] Obtaining a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node;

[0201] For each Redis node, determine the node type of the Redis node recognized by each Redis node according to each node information set; the node type includes at least one of being in the Redis cluster, not being in the Redis cluster, a master node, and a standby node;

[0202] A fault detection result of the Redis cluster is determined according to the number of first nodes of the Redis nodes that are identified as the same node type.

[0203] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0204] If the number of the first nodes is greater than or equal to the first preset threshold, the same node type is used as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; the first preset threshold is determined according to the total number of Redis nodes in the Redis cluster;

[0205] If the number of first nodes of the same node type is equal to half of the total number, the fault detection result is determined to be a Redis cluster fault.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0207] When it is determined that the fault detection result shows that the Redis cluster is normal, the number of second nodes of the Redis nodes whose target type is the primary node and which are deployed on a single device is obtained;

[0208] If the number of the second nodes is greater than the second preset threshold, the Redis node whose target type is the master node on the single device is migrated.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] Determine the node to be migrated from the Redis nodes whose target type is the master node;

[0211] Determine the number of third nodes of the master node on the target device where the standby node corresponding to the node to be migrated is located;

[0212] If the third number of nodes is less than the second preset threshold, the node to be migrated is migrated to the target device.

[0213] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0214] When it is determined that the fault detection result is a Redis cluster fault, obtaining a storage location and a storage method of first resource data of the first control environment from a resource data table;

[0215] According to the storage location and storage method, the first resource data is acquired and stored in the second control environment; the second control environment is a backup control environment for the first control environment.

[0216] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0217] When it is determined that the fault detection result is a Redis cluster fault, obtaining first business data of a first business process in a first control environment from a state data table;

[0218] The first business process is restored in the second control environment according to the first business data.

[0219] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0220] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.< / port:role:state:slaveip:slaveport>

Claims

1. A fault detection method, characterized in that: The method comprises: Obtaining a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node; For each of the Redis nodes, determine the node type of the Redis node recognized by each of the Redis nodes according to each of the node information sets; the node type includes at least one of being in the Redis cluster, not being in the Redis cluster, a primary node, and a standby node; A fault detection result of the Redis cluster is determined according to the number of first nodes of the Redis nodes that are identified as the same node type.

2. The method according to claim 1, characterized in that The determining, according to the number of the Redis nodes that are identified as the first nodes of the same node type, a fault detection result of the Redis cluster includes: If the first number of nodes is greater than or equal to a first preset threshold, the same node type is used as the target type of the Redis node, and the fault detection result is determined to be that the Redis cluster is normal; the first preset threshold is determined according to the total number of Redis nodes in the Redis cluster; If the number of first nodes of the same node type is equal to half of the total number, it is determined that the fault detection result is a Redis cluster fault.

3. The method according to claim 2, characterized in that The method further comprises: When it is determined that the fault detection result is that the Redis cluster is normal, obtaining the number of second nodes of the Redis nodes whose target type is the master node deployed on a single device; If the number of the second nodes is greater than a second preset threshold, the Redis node whose target type is a master node on the single device is migrated.

4. The method according to claim 3, characterized in that The migrating of the Redis node whose target type is a master node on the single device includes: Determine the node to be migrated from the Redis nodes whose target type is the master node; Determine the number of third nodes of the master node of the target device where the standby node corresponding to the node to be migrated is located; If the third number of nodes is less than the second preset threshold, the node to be migrated is migrated to the target device.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: When it is determined that the fault detection result is a fault of the Redis cluster, obtaining a storage location and a storage method of the first resource data of the first control environment from a resource data table; According to the storage location and the storage method, the first resource data is acquired and stored in a second control environment; the second control environment is a backup control environment for the first control environment.

6. The method according to claim 5, characterized in that The method further comprises: When it is determined that the fault detection result is a fault of the Redis cluster, obtaining first business data of the first business process in the first control environment from a state data table; The first business process is restored in the second control environment according to the first business data.

7. A fault detection device, characterized in that: The device comprises: A first acquisition module is used to acquire a node information set of each Redis node in a remote dictionary service Redis cluster in a first control environment; the node information set includes node information of each Redis node; A first determination module is used to determine, for each of the Redis nodes, a node type recognized by each of the Redis nodes according to each of the node information sets; the node type includes at least one of being in the Redis cluster, not being in the Redis cluster, a primary node, and a standby node; The second determination module is used to determine the fault detection result of the Redis cluster according to the number of the Redis nodes that are identified as the first nodes of the same node type.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images