Fault processing method and device for Redis cluster

Abstract

The invention discloses a fault processing method and device for a Redis cluster. The method comprises the steps that a plurality of first alarm arrays corresponding to a plurality of Redis clusters associated with a service system are acquired, and each array comprises triggering states of various alarm events of one Redis cluster; determining a fault Redis cluster and a processing priority according to the first alarm array; each fault cluster is sequentially processed according to the priority, specifically, a second alarm array of a host where the cluster is located is obtained, and the array records trigger states of various alarm events of the host; finding a target fault type matched with the first alarm array and the second alarm array from a pre-configured fault processing flow table, and executing a corresponding target processing strategy; the fault processing flow table stores a mapping relationship between the combination of various first alarm arrays and second alarm arrays and fault types and a processing strategy of each fault type. According to the method and the device, the technical problem of relatively low efficiency of a traditional mode of manually processing the Redis cluster fault is solved.

Description

Technical Field

[0001] The present application relates to the technical field of business system operation and maintenance, and in particular to a fault handling method and device for a Redis cluster. Background Art

[0002] CRM (Customer Relationship Management) is the core system of the enterprise. In the new generation CRM architecture of BSS4.0, Redis is used to store a variety of cache data including session, order information, process center data, MSOP (Market Service Operation Platform), SAOP (Service Activation and Operation Platform) and configuration. These data run through multiple subsystems and modules and are distributed in different Redis clusters. In particular, some key business scenarios are highly dependent on Redis. Once Redis fails, it will directly affect the normal operation of the core business, resulting in customers being unable to handle business and possibly causing complaints, which will have a negative impact on the corporate image. Traditionally, Redis fault handling relies on manual intervention by operation and maintenance personnel to restore services after receiving alarms, but this brings a series of challenges. First, it is difficult to quickly determine the processing priority when multiple Redis clusters fail at the same time. Second, manual analysis and problem solving take a long time and are inefficient. Therefore, in the information-based innovation environment, it is particularly important to design an effective Redis fault handling method and device based on the characteristics of the customer relationship management system, so as to improve system stability and response speed and ensure business continuity.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for handling faults of a Redis cluster, so as to at least solve the technical problem that the traditional method of manually handling faults of a Redis cluster is inefficient.

[0005] According to one aspect of an embodiment of the present application, a fault handling method for a Redis cluster is provided, comprising: obtaining multiple first alarm arrays corresponding to multiple Redis clusters associated with a business system, wherein each first alarm array includes triggering states of multiple alarm events corresponding to a Redis cluster; determining a faulty Redis cluster among the multiple Redis clusters based on the first alarm array, and determining a processing priority of each faulty Redis cluster; and executing the following fault handling process for each faulty Redis cluster in turn based on the processing priority: obtaining a second alarm array corresponding to a host where the faulty Redis cluster is located, wherein the second alarm array includes triggering states of multiple alarm events corresponding to the host; determining a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault handling flow table, determining a target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault handling flow table stores mapping relationships between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

[0006] Optionally, each element in the first alarm array corresponds to the triggering state of an alarm event of a Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: abnormal cluster state, loss of connection with the master node, space usage higher than a threshold, loss of connection with the slave node, master-slave data synchronization failure, and the number of node connections higher than a threshold; each element in the second alarm array corresponds to the triggering state of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

[0007] Optionally, determining a faulty Redis cluster among multiple Redis clusters based on the first alarm array includes: for each Redis cluster, when the first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, determining that the Redis cluster is a normal Redis cluster; when the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, determining that the Redis cluster is a faulty Redis cluster.

[0008] Optionally, determining the processing priority of each faulty Redis cluster includes: for each faulty Redis cluster, determining each alarm event triggered by the faulty Redis cluster according to the first alarm array corresponding to the faulty Redis cluster, and determining a preset alarm level corresponding to each triggered alarm event; summing the preset alarm levels corresponding to each triggered alarm event, and multiplying the obtained sum by a preset importance coefficient corresponding to the faulty Redis cluster to obtain a priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; and sorting each faulty Redis cluster from large to small according to the priority score to obtain the processing priority of each faulty Redis cluster.

[0009] Optionally, the fault handling flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a survival time; the matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; the processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, and the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; the survival time decays with the existence time of the pre-configuration item, when the survival time decays to 0, the pre-configuration item is automatically deleted, and when the pre-configuration item is matched by the faulty Redis cluster, the survival time is automatically reset.

[0010] Optionally, a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host is determined from a preconfigured fault handling flow table, including: for each preconfiguration item in the fault handling flow table, comparing the first alarm array corresponding to the preconfiguration item with the elements at the same position in the first alarm array corresponding to the faulty Redis cluster one by one, and determining a first ratio of the number of identical elements to the total number of elements in the first alarm array; comparing the second alarm array corresponding to the preconfiguration item with the elements at the same position in the second alarm array corresponding to the host where the faulty Redis cluster is located one by one, and determining a second ratio of the number of identical elements to the total number of elements in the second alarm array; performing weighted summation of the first ratio and the second ratio using a preset weight coefficient to obtain a matching degree between the preconfiguration item and the faulty Redis cluster; sorting the matching degrees corresponding to the preconfiguration items in the fault handling flow table from large to small, and when the maximum matching degree is not less than a preset threshold, determining that the fault type corresponding to the preconfiguration item corresponding to the maximum matching degree is the target fault type of the faulty Redis cluster.

[0011] Optionally, the method also includes: when the maximum matching degree is less than a preset threshold, processing the faulty Redis cluster according to a preset default processing strategy; generating a first prompt message, the first prompt message being used to prompt the target object to create a pre-configured item corresponding to a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

[0012] Optionally, after executing the target processing strategy, the method also includes: obtaining the first alarm array corresponding to the faulty Redis cluster again; when the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, generating a second prompt information, the second prompt information being used to prompt the target object to manually process the faulty Redis cluster, and modifying the processing strategy in the pre-configuration item corresponding to the faulty Redis cluster in the fault processing flow table.

[0013] According to another aspect of an embodiment of the present application, a fault handling device for a Redis cluster is also provided, including: an acquisition module, used to acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with a business system, wherein each first alarm array includes the trigger status of multiple alarm events corresponding to a Redis cluster; a determination module, used to determine the faulty Redis cluster among the multiple Redis clusters according to the first alarm array, and determine the processing priority of each faulty Redis cluster; a processing module, used to execute the following fault handling process for each faulty Redis cluster in turn according to the processing priority: acquire a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the trigger status of multiple alarm events corresponding to the host; determine a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault handling flow table, determine a target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault handling flow table stores a mapping relationship between a combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product comprising: a computer program, wherein when the computer program is executed by a processor, the above-mentioned Redis cluster fault handling method is implemented.

[0015] According to another aspect of an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned Redis cluster fault handling method through the computer program.

[0016] In an embodiment of the present application, by monitoring and collecting real-time alarm information of multiple Redis clusters associated with the business system and converting it into a first alarm array, the health status of the Redis cluster can be fully and instantly grasped; based on the first alarm array, it is possible to accurately identify which Redis clusters are in a faulty state, and by calculating the fault handling priority index and combining the importance coefficient of the cluster, the order of fault handling is determined. This method overcomes the problem of lack of priority sorting in traditional fault handling, ensures that the Redis cluster failure of the core business is handled with priority, and reduces the business interruption time; after the faulty Redis cluster is determined, the second alarm array of the host where it is located is further obtained, which introduces the fault information at the host level and improves the accuracy of fault location, because the failure of the Redis cluster may be caused by the hardware or system problems of the host; the fault handling flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type, realizing the automation and intelligence of fault handling. The operation and maintenance personnel do not need to manually analyze each alarm event, and can automatically match the fault type and execute the corresponding processing strategy, which greatly shortens the time of fault response and resolution, thereby solving the technical problem of low efficiency of the traditional manual processing of Redis cluster failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a flowchart of an optional Redis cluster fault handling method according to an embodiment of the present application;

[0019] Figure 2 It is a structural diagram of an optional Redis cluster fault handling device according to an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] Example 1

[0024] According to an embodiment of the present application, a fault handling method for a Redis cluster is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] Figure 1 is a flow chart of a Redis cluster fault handling method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0026] Step S102, obtaining multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each first alarm array includes triggering states of multiple alarm events corresponding to one Redis cluster;

[0027] Step S104, determining a faulty Redis cluster among the multiple Redis clusters according to the first alarm array, and determining a processing priority of each faulty Redis cluster;

[0028] Step S106, execute the following fault handling process for each faulty Redis cluster in turn according to the processing priority: obtain the second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the triggering status of multiple alarm events corresponding to the host; determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the preconfigured fault handling flow table, determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault handling flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

[0029] The following describes the steps of the Redis cluster troubleshooting method in combination with the specific implementation process.

[0030] The Redis cluster management platform can integrate automated fault detection and handling functions. The platform can achieve efficient management of Redis cluster faults through real-time monitoring, intelligent alarm analysis, fault priority scheduling, automated fault handling and continuous optimization mechanisms. Specifically, the platform can perform the following methods and steps:

[0031] First, multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system are obtained, wherein each first alarm array includes triggering states of multiple alarm events corresponding to a Redis cluster.

[0032] As an optional implementation, each element in the first alarm array corresponds to the triggering status of an alarm event of the Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: abnormal cluster status, loss of connection with the master node, space utilization rate higher than a threshold, loss of connection with the slave node, master-slave data synchronization failure, and the number of node connections higher than a threshold.

[0033] The first alarm array of each Redis cluster consists of multiple binary bits, each of which represents whether an alarm event is triggered. For example, if the first bit of the array is 1, it means that the cluster status abnormal alarm event is triggered; if the second bit is 1, it means that the master node disconnection alarm event is triggered; and so on. If an alarm event is not triggered, the corresponding array bit is 0.

[0034] In addition, the severity levels of alarm events can be further differentiated. For example, different numerical values ​​(such as 0 not triggered, 1 normal alarm, 2 severe alarm) are used to indicate the triggering status of alarm events, which helps to more accurately evaluate the priority of fault handling in subsequent steps.

[0035] After obtaining the first alarm data, a faulty Redis cluster among the multiple Redis clusters is determined according to the first alarm array, and a processing priority of each faulty Redis cluster is determined.

[0036] As an optional implementation, a faulty Redis cluster among multiple Redis clusters is determined based on the first alarm array. The process can take the following steps: for each Redis cluster, when the first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, the Redis cluster is determined to be a normal Redis cluster; when the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, the Redis cluster is determined to be a faulty Redis cluster.

[0037] It can be understood as reading the first alarm array of each Redis cluster provided by the alarm docking module. The array contains the trigger status of all current alarm events of the cluster. For each element in the array, check whether its value is 1. If there is any element with a value of 1 in the array, it means that at least one alarm event is triggered, thereby determining that the Redis cluster is a faulty Redis cluster. If the values ​​of all elements in the array are 0, it means that the cluster currently has not triggered any alarm event, and the system marks it as a normal Redis cluster; otherwise, it is marked as a faulty Redis cluster.

[0038] As an optional implementation, the processing priority of each faulty Redis cluster is determined. The process may take the following steps: for each faulty Redis cluster, determine each alarm event triggered by the faulty Redis cluster according to the first alarm array corresponding to the faulty Redis cluster, and determine the preset alarm level corresponding to each triggered alarm event; sum the preset alarm levels corresponding to each triggered alarm event, and multiply the sum by the preset importance coefficient corresponding to the faulty Redis cluster to obtain the priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; sort each faulty Redis cluster from large to small according to the priority score to obtain the processing priority of each faulty Redis cluster.

[0039] The calculation formula of the priority score can be expressed as follows:

[0040] U i =[u i1 ,u i2 ,...,u im ](u ij∈{0,1,2},j=1,2,3,...,)

[0041]

[0042] PR i = fun(U i )*α i

[0043] Where U i Indicates the first alarm array corresponding to the i-th faulty Redis cluster, u ij represents the alarm level corresponding to the jth alarm event in the first alarm array corresponding to the i-th faulty Redis cluster, m represents the number of alarm events in the first alarm array corresponding to the i-th faulty Redis cluster, fun(U i ) represents the sum of the preset alarm levels corresponding to the alarm events in the first alarm array corresponding to the i-th faulty Redis cluster, α i Represents the preset importance coefficient corresponding to the i-th faulty Redis cluster, PR i Indicates the priority score corresponding to the i-th faulty Redis cluster.

[0044] It can be understood that for a Redis cluster identified as a fault, the system needs to further analyze the preset alarm levels corresponding to the alarm events triggered in its first alarm array. The preset alarm levels can be set based on the severity and urgency of the alarm events. For example, cluster status abnormality and master node loss of connection may be set as severe alarm levels. The preset alarm levels of the triggered alarm events are summed to obtain the alarm level and value of the faulty Redis cluster. This sum reflects the comprehensive severity of the current alarm events of the cluster. In order to further quantify the processing priority of the faulty Redis cluster, the system multiplies the alarm level and value by the preset importance coefficient corresponding to the faulty Redis cluster. This coefficient is set according to the importance of the business scenario in which the cluster is located and the impact of the cluster failure on the business. For example, the importance coefficient of the cluster involving core business will be higher. The result of the above calculation is the priority score of the faulty Redis cluster, which combines the severity of the alarm event and the importance of the cluster, and is a key indicator for determining the order of fault handling.

[0045] After obtaining the processing priority, the following fault handling process is performed on each faulty Redis cluster in turn according to the processing priority:

[0046] Obtain a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes triggering states of multiple alarm events corresponding to the host;

[0047] Determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the preconfigured fault processing flow table, determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault processing flow table stores mapping relationships between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as processing strategies corresponding to each fault type.

[0048] Among them, as an optional implementation method, each element in the second alarm array corresponds to the triggering status of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

[0049] In the process of executing the implementation steps of the Redis cluster management platform, it is necessary to combine with the platform database. The platform database stores configuration information closely related to fault handling, including the connection information of the Redis cluster and the host, the importance coefficient of the cluster, and the fault handling flow table. This information forms the basis for automatic fault handling, ensuring that the system can respond quickly and take effective measures when a fault occurs.

[0050] As an optional implementation, the fault handling flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a survival time; the matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; the processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, and the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; the survival time decays with the existence time of the pre-configuration item, and when the survival time decays to 0, the pre-configuration item is automatically deleted, and when the pre-configuration item is matched by the faulty Redis cluster, the survival time is automatically reset.

[0051] As an optional implementation, a target fault type matching the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host is determined from a preconfigured fault processing flow table. The process may be performed in the following steps:

[0052] S1, for each pre-configured item in the fault processing flow table, compare the elements at the same position in the first alarm array corresponding to the pre-configured item with the elements in the first alarm array corresponding to the faulty Redis cluster one by one, and determine a first ratio of the number of identical elements to the total number of elements in the first alarm array;

[0053] The calculation formula of the ratio can be expressed as:

[0054]

[0055] In the formula, a j ' represents the jth element in the first alarm array corresponding to the pre-configuration item, a ij represents the jth element in the first alarm array corresponding to the i-th faulty Redis cluster, δ(a j ′,a ij ) is a comparison function (or indicator function). j ′ =a ij , returns 1, otherwise 0, m is the total number of elements in the first alarm array, k iA Indicates the first ratio of the number of equal elements of the two arrays at all positions to the total number of elements in the first alarm array.

[0056] S2, comparing the elements at the same position in the second alarm array corresponding to the pre-configuration item and the second alarm array corresponding to the host where the faulty Redis cluster is located one by one, and determining a second ratio of the number of the same elements to the total number of elements in the second alarm array;

[0057] The calculation formula of the ratio can be expressed as:

[0058]

[0059] Where b j ' represents the jth element in the second alarm array corresponding to the pre-configuration item, b ij represents the jth element in the second alarm array corresponding to the host where the i-th faulty Redis cluster is located, δ(b j ′,b ij ) is a comparison function (or indicator function), when (b j ′ =b ij ), returns 1, otherwise 0, n represents the total number of elements in the second alarm array, k iB A first ratio of the number of equal elements of the two arrays at all positions to the total number of elements of the second alarm array.

[0060] S3, performing weighted summation of the first ratio and the second ratio using a preset weight coefficient to obtain a matching degree between the pre-configured item and the faulty Redis cluster;

[0061] S4, sorting the matching degrees corresponding to the pre-configuration items in the fault processing flow table from large to small, and when the maximum matching degree is not less than a preset threshold, determining that the fault type corresponding to the pre-configuration item corresponding to the maximum matching degree is the target fault type of the faulty Redis cluster.

[0062] Once the target fault type is determined, specific operations will be performed according to the processing strategy corresponding to the fault type in the fault processing flow table, such as adjusting Redis configuration parameters, restarting services, and migrating to a standby cluster.

[0063] As an optional implementation, the following steps may be taken for optimization: when the maximum matching degree is less than a preset threshold, the faulty Redis cluster is processed according to a preset default processing strategy; a first prompt message is generated, and the first prompt message is used to prompt the target object to create a pre-configured item corresponding to a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

[0064] As an optional implementation, after executing the target processing strategy, a feedback and optimization process can also be executed, including: obtaining the first alarm array corresponding to the faulty Redis cluster again; when the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, generating a second prompt information, the second prompt information is used to prompt the target object to manually process the faulty Redis cluster, and modify the processing strategy in the pre-configured item corresponding to the faulty Redis cluster in the fault processing flow table.

[0065] In an embodiment of the present application, by monitoring and collecting real-time alarm information of multiple Redis clusters associated with the business system and converting it into a first alarm array, the health status of the Redis cluster can be fully and instantly grasped; based on the first alarm array, it is possible to accurately identify which Redis clusters are in a faulty state, and by calculating the fault handling priority index and combining the importance coefficient of the cluster, the order of fault handling is determined. This method overcomes the problem of lack of priority sorting in traditional fault handling, ensures that the Redis cluster failure of the core business is handled with priority, and reduces the business interruption time; after the faulty Redis cluster is determined, the second alarm array of the host where it is located is further obtained, which introduces the fault information at the host level and improves the accuracy of fault location, because the failure of the Redis cluster may be caused by the hardware or system problems of the host; the fault handling flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type, realizing the automation and intelligence of fault handling. The operation and maintenance personnel do not need to manually analyze each alarm event, and can automatically match the fault type and execute the corresponding processing strategy, which greatly shortens the time of fault response and resolution, thereby solving the technical problem of low efficiency of the traditional manual processing of Redis cluster failures.

[0066] Example 2

[0067] According to an embodiment of the present application, a fault handling device for a Redis cluster is provided for implementing the fault handling method for a Redis cluster in Embodiment 1. Figure 2 As shown, the fault handling device of the Redis cluster at least includes: an acquisition module 21, a determination module 22 and a processing module 23, wherein:

[0068] An acquisition module 21 is used to acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each first alarm array includes triggering states of multiple alarm events corresponding to one Redis cluster;

[0069] A determination module 22, configured to determine a faulty Redis cluster among the multiple Redis clusters according to the first alarm array, and determine a processing priority of each faulty Redis cluster;

[0070] The processing module 23 is used to execute the following fault handling process for each faulty Redis cluster in turn according to the processing priority: obtain the second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the trigger status of multiple alarm events corresponding to the host; determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the pre-configured fault handling flow table, and determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault handling flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

[0071] The functions of each module of the fault handling device of the Redis cluster are explained below in conjunction with a specific implementation process.

[0072] The acquisition module acquires multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each first alarm array includes triggering states of multiple alarm events corresponding to one Redis cluster.

[0073] As an optional implementation, each element in the first alarm array corresponds to the triggering status of an alarm event of the Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: abnormal cluster status, loss of connection with the master node, space utilization rate higher than a threshold, loss of connection with the slave node, master-slave data synchronization failure, and the number of node connections higher than a threshold.

[0074] After obtaining the first alarm data, the determination module determines the faulty Redis cluster among the multiple Redis clusters according to the first alarm array, and determines the processing priority of each faulty Redis cluster.

[0075] As an optional implementation, a faulty Redis cluster among multiple Redis clusters is determined based on the first alarm array. The process can take the following steps: for each Redis cluster, when the first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, the Redis cluster is determined to be a normal Redis cluster; when the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, the Redis cluster is determined to be a faulty Redis cluster.

[0076] As an optional implementation, the processing priority of each faulty Redis cluster is determined. The process may take the following steps: for each faulty Redis cluster, determine each alarm event triggered by the faulty Redis cluster according to the first alarm array corresponding to the faulty Redis cluster, and determine the preset alarm level corresponding to each triggered alarm event; sum the preset alarm levels corresponding to each triggered alarm event, and multiply the sum by the preset importance coefficient corresponding to the faulty Redis cluster to obtain the priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; sort each faulty Redis cluster from large to small according to the priority score to obtain the processing priority of each faulty Redis cluster.

[0077] After determining the priority, the processing module executes the following fault handling process for each faulty Redis cluster in turn according to the processing priority:

[0078] Obtain a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes triggering states of multiple alarm events corresponding to the host;

[0079] Determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the preconfigured fault processing flow table, determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault processing flow table stores mapping relationships between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as processing strategies corresponding to each fault type.

[0080] As an optional implementation, each element in the second alarm array corresponds to the triggering status of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

[0081] As an optional implementation, the fault handling flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a survival time; the matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; the processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, and the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; the survival time decays with the existence time of the pre-configuration item, and when the survival time decays to 0, the pre-configuration item is automatically deleted, and when the pre-configuration item is matched by the faulty Redis cluster, the survival time is automatically reset.

[0082] As an optional implementation, a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host is determined from a preconfigured fault processing flow table. The process can be carried out in the following steps: for each preconfigured item in the fault processing flow table, the first alarm array corresponding to the preconfigured item is compared with the elements at the same position in the first alarm array corresponding to the faulty Redis cluster one by one, and a first ratio of the number of identical elements to the total number of elements in the first alarm array is determined; the second alarm array corresponding to the preconfigured item is compared with the elements at the same position in the second alarm array corresponding to the host where the faulty Redis cluster is located, and a second ratio of the number of identical elements to the total number of elements in the second alarm array is determined; the first ratio and the second ratio are weightedly summed using a preset weight coefficient to obtain a matching degree between the preconfigured item and the faulty Redis cluster; the matching degrees corresponding to the preconfigured items in the fault processing flow table are sorted from large to small, and when the maximum matching degree is not less than a preset threshold, the fault type corresponding to the preconfigured item corresponding to the maximum matching degree is determined to be the target fault type of the faulty Redis cluster.

[0083] As an optional implementation, the following steps may be taken for optimization: when the maximum matching degree is less than a preset threshold, the faulty Redis cluster is processed according to a preset default processing strategy; a first prompt message is generated, and the first prompt message is used to prompt the target object to create a pre-configured item corresponding to a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

[0084] As an optional implementation, after executing the target processing strategy, the method also includes: obtaining the first alarm array corresponding to the faulty Redis cluster again; when the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, generating second prompt information, the second prompt information is used to prompt the target object to manually process the faulty Redis cluster, and modify the processing strategy in the pre-configuration item corresponding to the faulty Redis cluster in the fault processing flow table.

[0085] It should be noted that each module in the fault handling device for the Redis cluster in the embodiment of the present application corresponds one-to-one to each implementation step of the fault handling method for the Redis cluster in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1, and will not be repeated here.

[0086] Example 3

[0087] According to an embodiment of the present application, a computer program product is further provided. The computer program product includes a computer program. When the computer program is executed by a processor, the fault handling method of the Redis cluster in Embodiment 1 is implemented.

[0088] According to an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the Redis cluster fault handling method in Example 1 by running the computer program.

[0089] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the fault handling method of the Redis cluster in Example 1 is executed when the computer program is running.

[0090] According to an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the Redis cluster fault handling method in Example 1 through the computer program.

[0091] Specifically, when the computer program is running, the following steps are executed: obtaining multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each first alarm array includes the triggering status of multiple alarm events corresponding to a Redis cluster; determining the faulty Redis cluster among the multiple Redis clusters based on the first alarm array, and determining the processing priority of each faulty Redis cluster; executing the following fault processing flow for each faulty Redis cluster in turn based on the processing priority: obtaining the second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the triggering status of multiple alarm events corresponding to the host; determining the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, and determining the target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault processing flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

[0092] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 The hardware structure block diagram of an electronic device for implementing a fault handling method of a Redis cluster is shown. Figure 3 As shown, the electronic device 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0093] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0094] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the fault handling method of the Redis cluster in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the electronic device 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The transmission device 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 30. In one example, the transmission device 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0096] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 30 .

[0097] The serial numbers of the above embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0103] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A fault handling method for a Redis cluster, characterized in that: include: Acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each of the first alarm arrays includes triggering states of multiple alarm events corresponding to one Redis cluster; Determine a faulty Redis cluster among multiple Redis clusters according to the first alarm array, and determine a processing priority of each of the faulty Redis clusters; The following fault handling process is performed on each faulty Redis cluster in turn according to the processing priority: Obtain a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes triggering states of multiple alarm events corresponding to the host; Determine a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, determine a target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault processing flow table stores mapping relationships between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

2. The method according to claim 1, characterized in that Each element in the first alarm array corresponds to a triggering state of an alarm event of the Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: abnormal cluster state, master node disconnection, space usage rate higher than a threshold, slave node disconnection, master-slave data synchronization failure, and the number of node connections higher than a threshold; Each element in the second alarm array corresponds to the triggering status of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

3. The method according to claim 1, characterized in that Determining a faulty Redis cluster among multiple Redis clusters according to the first alarm array includes: For each Redis cluster, when a first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, determining that the Redis cluster is a normal Redis cluster; When the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, it is determined that the Redis cluster is a faulty Redis cluster.

4. The method according to claim 1, characterized in that: Determine the processing priority of each of the faulty Redis clusters, including: For each faulty Redis cluster, determine each alarm event triggered by the faulty Redis cluster according to the first alarm array corresponding to the faulty Redis cluster, and determine the preset alarm level corresponding to each triggered alarm event; The preset alarm levels corresponding to the triggered alarm events are summed, and the obtained sum is multiplied by the preset importance coefficient corresponding to the faulty Redis cluster to obtain the priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; The faulty Redis clusters are sorted from large to small according to the priority scores to obtain the processing priority of each faulty Redis cluster.

5. The method according to claim 2, characterized in that: The fault processing flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a survival time; The matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; The processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, wherein the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; The survival time decays with the existence time of the pre-configuration item. When the survival time decays to 0, the pre-configuration item is automatically deleted. When the pre-configuration item is matched by a faulty Redis cluster, the survival time is automatically reset.

6. The method according to claim 5, characterized in that Determining a target fault type that matches a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to the host from a preconfigured fault processing flow table includes: For each pre-configuration item in the fault processing flow table, compare the elements at the same position in the first alarm array corresponding to the pre-configuration item and the first alarm array corresponding to the faulty Redis cluster one by one, and determine a first ratio of the number of identical elements to the total number of elements in the first alarm array; Compare the elements at the same position in the second alarm array corresponding to the pre-configuration item and the second alarm array corresponding to the host where the faulty Redis cluster is located one by one, and determine a second ratio of the number of the same elements to the total number of elements in the second alarm array; Performing a weighted summation of the first ratio and the second ratio using a preset weight coefficient to obtain a matching degree between the preconfigured item and the faulty Redis cluster; The matching degrees corresponding to the pre-configuration items in the fault processing flow table are sorted from large to small. When the maximum matching degree is not less than a preset threshold, the fault type corresponding to the pre-configuration item corresponding to the maximum matching degree is determined to be the target fault type of the faulty Redis cluster.

7. The method according to claim 6, characterized in that The method further comprises: When the maximum matching degree is less than the preset threshold, the faulty Redis cluster is processed according to a preset default processing strategy; Generate first prompt information, where the first prompt information is used to prompt the target object to create pre-configured items corresponding to a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to a host where the faulty Redis cluster is located in the fault processing flow table.

8. The method according to claim 5, characterized in that After executing the target processing strategy, the method further includes: Obtain the first alarm array corresponding to the faulty Redis cluster again; When the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, second prompt information is generated, and the second prompt information is used to prompt the target object to manually process the faulty Redis cluster and modify the processing strategy in the pre-configuration item corresponding to the faulty Redis cluster in the fault processing flow table.

9. A fault handling device for a Redis cluster, characterized in that: include: An acquisition module, used to acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each of the first alarm arrays includes triggering states of multiple alarm events corresponding to a Redis cluster; A determination module, used to determine a faulty Redis cluster among multiple Redis clusters according to the first alarm array, and determine a processing priority of each of the faulty Redis clusters; A processing module is used to execute the following fault handling process for each of the faulty Redis clusters in turn according to the processing priority: obtaining a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the triggering status of multiple alarm events corresponding to the host; determining a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault handling flow table, and determining a target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault handling flow table stores a mapping relationship between a combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

10. An electronic device, characterized in that: include: A memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the Redis cluster fault handling method according to any one of claims 1 to 8 through the computer program.

Images