Fault handling method and device for redis cluster

Abstract

The application discloses a Redis cluster fault processing method and device. The method comprises the following steps: acquiring a plurality of first alarm arrays corresponding to a plurality of Redis clusters associated with a business system, each array containing a triggering state of a plurality of alarm events of a Redis cluster; determining a fault Redis cluster and a processing priority according to the first alarm array; processing each fault cluster in turn according to the priority, specifically: acquiring a second alarm array of a host where the cluster is located, the array recording a triggering state of a plurality of alarm events of the host; finding a target fault type matched with the first alarm array and the second alarm array from a preconfigured fault processing flow table, and executing a corresponding target processing strategy; and the fault processing flow table stores a mapping relationship between a plurality of combinations of the first alarm array and the second alarm array and a fault type and a processing strategy of each fault type. The application solves the technical problem of low efficiency of a traditional manual Redis cluster fault processing mode.

Description

Technical Field

[0001] The present application relates to the technical field of business system operation and maintenance, and more specifically, to a Redis cluster fault handling method and device. Background Art

[0002] As a core enterprise system, the Customer Relationship Management (CRM) system uses Redis in the next-generation BSS4.0 CRM architecture to store a variety of cached data, including sessions, order information, process-centric data, the Market Service Operation Platform (MSOP), the Service Activation and Operation Platform (SAOP), and configurations. This data spans multiple subsystems and modules and is distributed across different Redis clusters. Certain critical business scenarios are highly dependent on Redis. A Redis failure would directly impact the normal operation of core business operations, preventing customers from transacting, potentially leading to complaints and negatively impacting the company's image. Traditionally, Redis failure handling relies on manual intervention by operations personnel upon receiving alerts to restore service, but this presents a number of challenges. First, when multiple Redis clusters fail simultaneously, it's difficult to quickly prioritize. Second, manual analysis and resolution of issues is time-consuming and inefficient. Therefore, in the information-based innovation environment, it is particularly important to design an effective Redis fault handling method and device based on the characteristics of the customer relationship management system, so as to improve system stability and response speed and ensure business continuity.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method and apparatus for handling Redis cluster faults, so as to at least solve the technical problem of low efficiency of the traditional manual handling of Redis cluster faults.

[0005] According to an aspect of the embodiments of the present application, a fault processing method of a Redis cluster is provided, comprising: obtaining a plurality of first alarm arrays corresponding to a plurality of Redis clusters associated with a business system, wherein each first alarm array includes a triggering state of a plurality of alarm events corresponding to a Redis cluster; determining a fault Redis cluster in the plurality of Redis clusters according to the first alarm arrays, and determining a processing priority of each fault Redis cluster; sequentially performing the following fault processing procedure on each fault Redis cluster according to the processing priority: obtaining a second alarm array corresponding to a host where the fault Redis cluster is located, wherein the second alarm array includes a triggering state of a plurality of alarm events corresponding to the host; determining a target fault type matched with the first alarm array corresponding to the fault Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, and determining a target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault processing flow table stores a mapping relationship between a plurality of combinations of the first alarm arrays and the second alarm arrays and a plurality of fault types, and a processing strategy corresponding to each fault type.

[0006] Optionally, each element in the first alarm array corresponds to a triggering state of an alarm event of the Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: cluster state exception, master node disconnection, space usage rate higher than a threshold value, slave node disconnection, master-slave data synchronization failure, and node connection number higher than a threshold value; each element in the second alarm array corresponds to a triggering state of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold value, insufficient memory, insufficient disk space, and network load higher than a threshold value.

[0007] Optionally, determining a fault Redis cluster in the plurality of Redis clusters according to the first alarm arrays comprises: for each Redis cluster, in a case where the first alarm array corresponding to the Redis cluster indicates that none of the plurality of alarm events corresponding to the Redis cluster is triggered, determining that the Redis cluster is a normal Redis cluster; in a case where the first alarm array corresponding to the Redis cluster indicates that at least one of the plurality of alarm events corresponding to the Redis cluster is triggered, determining that the Redis cluster is a fault Redis cluster.

[0008] Optionally, determining the processing priority of each faulty Redis cluster includes: for each faulty Redis cluster, determining each alarm event triggered by the faulty Redis cluster based on the first alarm array corresponding to the faulty Redis cluster, and determining a preset alarm level corresponding to each triggered alarm event; summing the preset alarm levels corresponding to each triggered alarm event, and multiplying the obtained sum by a preset importance coefficient corresponding to the faulty Redis cluster to obtain a priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; and sorting each faulty Redis cluster from large to small according to the priority score to obtain the processing priority of each faulty Redis cluster.

[0009] Optionally, the fault handling flow table includes multiple pre-configuration items, each pre-configuration item includes: a matching domain, a processing strategy and a survival time; the matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; the processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, and the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; the survival time decays with the existence time of the pre-configuration item. When the survival time decays to 0, the pre-configuration item is automatically deleted. When the pre-configuration item is matched by the faulty Redis cluster, the survival time is automatically reset.

[0010] Optionally, a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host is determined from a preconfigured fault processing flow table, including: for each preconfigured item in the fault processing flow table, comparing the first alarm array corresponding to the preconfigured item with the elements at the same position in the first alarm array corresponding to the faulty Redis cluster one by one, and determining a first ratio of the number of identical elements to the total number of elements in the first alarm array; comparing the second alarm array corresponding to the preconfigured item with the elements at the same position in the second alarm array corresponding to the host where the faulty Redis cluster is located one by one, and determining a second ratio of the number of identical elements to the total number of elements in the second alarm array; using a preset weight coefficient to perform weighted summation on the first ratio and the second ratio to obtain a matching degree between the preconfigured item and the faulty Redis cluster; sorting the matching degrees corresponding to the preconfigured items in the fault processing flow table from large to small, and when the maximum matching degree is not less than a preset threshold, determining that the fault type corresponding to the preconfiguration item corresponding to the maximum matching degree is the target fault type of the faulty Redis cluster.

[0011] Optionally, the method also includes: when the maximum matching degree is less than a preset threshold, processing the faulty Redis cluster according to a preset default processing strategy; generating a first prompt message, the first prompt message being used to prompt the target object to create a pre-configured item corresponding to a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

[0012] Optionally, after executing the target processing strategy, the method further includes: obtaining the first alarm array corresponding to the faulty Redis cluster again; when the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, generating a second prompt message, the second prompt message being used to prompt the target object to manually process the faulty Redis cluster, and modify the processing strategy in the pre-configured item corresponding to the faulty Redis cluster in the fault processing flow table.

[0013] According to another aspect of an embodiment of the present application, a fault handling device for a Redis cluster is also provided, including: an acquisition module, used to obtain multiple first alarm arrays corresponding to multiple Redis clusters associated with a business system, wherein each first alarm array includes the trigger status of multiple alarm events corresponding to a Redis cluster; a determination module, used to determine the faulty Redis cluster among the multiple Redis clusters based on the first alarm array, and determine the processing priority of each faulty Redis cluster; a processing module, used to execute the following fault handling process for each faulty Redis cluster in sequence according to the processing priority: obtaining a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the trigger status of multiple alarm events corresponding to the host; determining a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault handling flow table, determining a target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault handling flow table stores a mapping relationship between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product including: a computer program, wherein when the computer program is executed by a processor, the above-mentioned Redis cluster fault handling method is implemented.

[0015] According to another aspect of an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned Redis cluster fault handling method through the computer program.

[0016] In an embodiment of the present application, by monitoring and collecting real-time alarm information from multiple Redis clusters associated with a business system and converting it into a first alarm array, the health status of the Redis cluster can be comprehensively and immediately understood. Based on the first alarm array, which Redis clusters are in a faulty state can be accurately identified, and the order of fault handling can be determined by calculating the fault handling priority index and combining it with the cluster's importance coefficient. This approach overcomes the lack of priority sorting in traditional fault handling, ensuring that Redis cluster faults in core businesses are handled first, thereby reducing business interruption time. After the faulty Redis cluster is determined, a second alarm array of the host on which it is located is further obtained. This introduces host-level fault information and improves the accuracy of fault location, because Redis cluster faults may be caused by hardware or system problems in the host. The fault handling flow table stores the mapping relationship between multiple combinations of the first alarm array and the second alarm array and multiple fault types, as well as the corresponding processing strategy for each fault type, realizing automated and intelligent fault handling. Operations and maintenance personnel do not need to manually analyze each alarm event, but can automatically match the fault type and execute the corresponding processing strategy, greatly shortening the time for fault response and resolution, thereby solving the technical problem of low efficiency of traditional manual handling of Redis cluster faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a flowchart of an optional Redis cluster fault handling method according to an embodiment of the present application;

[0019] Figure 2 This is a schematic structural diagram of an optional Redis cluster fault handling device according to an embodiment of the present application;

[0020] Figure 3 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0022] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0023] Example 1

[0024] According to an embodiment of the present application, a method for troubleshooting a Redis cluster is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown.

[0025] Figure 1 This is a flow chart of a Redis cluster fault handling method provided in accordance with an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0026] Step S102: Acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each first alarm array includes triggering states of multiple alarm events corresponding to one Redis cluster;

[0027] Step S104: determining a faulty Redis cluster among the multiple Redis clusters according to the first alarm array, and determining a processing priority for each faulty Redis cluster;

[0028] Step S106: Execute the following fault handling process for each faulty Redis cluster in turn according to the processing priority: obtain the second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the trigger status of multiple alarm events corresponding to the host; determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the pre-configured fault handling flow table, determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault handling flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

[0029] The following describes the steps of the Redis cluster troubleshooting method in combination with the specific implementation process.

[0030] The Redis cluster management platform can integrate automated fault detection and handling capabilities. Through real-time monitoring, intelligent alarm analysis, fault prioritization, automated fault handling, and continuous optimization mechanisms, the platform can achieve efficient management of Redis cluster faults. Specifically, the platform can perform the following methods and steps:

[0031] First, a plurality of first alarm arrays corresponding to a plurality of Redis clusters associated with the business system are obtained, wherein each first alarm array includes triggering states of a plurality of alarm events corresponding to a Redis cluster.

[0032] As an optional implementation, each element in the first alarm array corresponds to the triggering status of an alarm event of the Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: abnormal cluster status, loss of connection of the master node, space usage higher than a threshold, loss of connection of the slave node, master-slave data synchronization failure, and the number of node connections higher than a threshold.

[0033] The first alarm array of each Redis cluster consists of multiple binary bits, each of which represents whether an alarm event has been triggered. For example, if the first bit of the array is 1, it means that the cluster status abnormality alarm event has been triggered; if the second bit is 1, it means that the master node loss alarm event has been triggered, and so on. If an alarm event is not triggered, the corresponding array bit is 0.

[0034] In addition, the severity levels of alarm events can be further differentiated. For example, different numerical values ​​(such as 0 not triggered, 1 normal alarm, 2 severe alarm) are used to represent the trigger status of the alarm event, which helps to more accurately evaluate the priority of fault handling in subsequent steps.

[0035] After obtaining the first alarm data, a faulty Redis cluster among the multiple Redis clusters is determined according to the first alarm array, and a processing priority of each faulty Redis cluster is determined.

[0036] As an optional implementation, a faulty Redis cluster among multiple Redis clusters is determined based on the first alarm array. The process can take the following steps: for each Redis cluster, when the first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, the Redis cluster is determined to be a normal Redis cluster; when the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, the Redis cluster is determined to be a faulty Redis cluster.

[0037] This can be understood as reading the first alarm array for each Redis cluster provided by the alarm connection module. This array contains the trigger status of all current alarm events in the cluster. For each element in the array, check whether its value is 1. If any element in the array has a value of 1, it means that at least one alarm event has been triggered, and thus the Redis cluster is determined to be a faulty Redis cluster. If the values ​​of all elements in the array are 0, it means that the cluster has not triggered any alarm events and the system marks it as a normal Redis cluster; otherwise, it is marked as a faulty Redis cluster.

[0038] As an optional implementation method, the processing priority of each faulty Redis cluster is determined. The process can take the following steps: for each faulty Redis cluster, determine the various alarm events triggered by the faulty Redis cluster based on the first alarm array corresponding to the faulty Redis cluster, and determine the preset alarm level corresponding to each triggered alarm event; sum the preset alarm levels corresponding to each triggered alarm event, and multiply the obtained sum by the preset importance coefficient corresponding to the faulty Redis cluster to obtain the priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; sort the various faulty Redis clusters from large to small according to the priority score to obtain the processing priority of each faulty Redis cluster.

[0039] The calculation formula of the priority score can be expressed as follows:

[0040] U i =[u i1 ,u i2 ,...,u im ](u ij∈{0,1,2},j=1,2,3,...,)

[0041]

[0042] PR i =fun(U i )*α i

[0043] Where U i Indicates the first alarm array corresponding to the i-th fault Redis cluster, u ij represents the alarm level corresponding to the jth alarm event in the first alarm array corresponding to the i-th fault Redis cluster, m represents the number of alarm events in the first alarm array corresponding to the i-th fault Redis cluster, fun(U i ) represents the sum of the preset alarm levels corresponding to the alarm events in the first alarm array corresponding to the i-th fault Redis cluster, α i Indicates the preset importance coefficient corresponding to the i-th fault Redis cluster, PR i Indicates the priority score corresponding to the i-th faulty Redis cluster.

[0044] It can be understood that for a Redis cluster identified as faulty, the system needs to further analyze the preset alarm levels corresponding to the triggered alarm events in its first alarm array. The preset alarm levels can be set based on the severity and urgency of the alarm events. For example, cluster status abnormalities and master node loss may be set to severe alarm levels. The preset alarm levels of the triggered alarm events are summed to obtain the alarm level sum value of the faulty Redis cluster. This sum value reflects the comprehensive severity of the cluster's current alarm events. To further quantify the processing priority of the faulty Redis cluster, the system multiplies the alarm level sum value by the preset importance coefficient corresponding to the faulty Redis cluster. This coefficient is set based on the importance of the cluster's business scenario and the impact of the cluster failure on the business. For example, the importance coefficient of a cluster involving core business will be higher. The result of the above calculation is the priority score of the faulty Redis cluster. It combines the severity of the alarm event and the importance of the cluster and is a key indicator for determining the order of fault handling.

[0045] After obtaining the processing priority, the following fault handling process is performed on each faulty Redis cluster in turn according to the processing priority:

[0046] Obtain a second alarm array corresponding to the host where the faulty Redis cluster is located, where the second alarm array includes triggering statuses of multiple alarm events corresponding to the host;

[0047] Determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the preconfigured fault processing flow table, determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy. The fault processing flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

[0048] As an optional implementation, each element in the second alarm array corresponds to the triggering status of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

[0049] During the implementation steps of the Redis cluster management platform, it is necessary to integrate with the platform database. The platform database stores configuration information closely related to fault handling, including the connection information between the Redis cluster and the host, the cluster's importance coefficient, and the fault handling flow table. This information forms the basis for automated fault handling, ensuring that the system can respond quickly and take effective measures when a fault occurs.

[0050] As an optional implementation, the fault handling flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a lifetime; the matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; the processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, and the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; the lifetime decays with the existence time of the pre-configuration item, and when the lifetime decays to 0, the pre-configuration item is automatically deleted, and when the pre-configuration item is matched by the faulty Redis cluster, the lifetime is automatically reset.

[0051] As an optional implementation, a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host is determined from a preconfigured fault handling flow table. This process can be performed in the following steps:

[0052] S1, for each pre-configured item in the fault handling flow table, compare the first alarm array corresponding to the pre-configured item with the elements at the same position in the first alarm array corresponding to the faulty Redis cluster one by one, and determine a first ratio of the number of identical elements to the total number of elements in the first alarm array;

[0053] The calculation formula of the ratio can be expressed as:

[0054]

[0055] Where a j ' represents the jth element in the first alarm array corresponding to the pre-configured item, a ij Represents the jth element in the first alarm array corresponding to the i-th fault Redis cluster, δ(a j ′,a ij ) is a comparison function (or indicator function), when a j ′ =a ij , returns 1, otherwise 0, m is the total number of elements in the first alarm array, k iA Indicates the first ratio of the number of equal elements in the two arrays at all positions to the total number of elements in the first alarm array.

[0056] S2: Compare the elements at the same position in the second alarm array corresponding to the pre-configured item with those in the second alarm array corresponding to the host where the faulty Redis cluster is located, one by one, to determine a second ratio of the number of identical elements to the total number of elements in the second alarm array;

[0057] The calculation formula of the ratio can be expressed as:

[0058]

[0059] Where b j ' represents the jth element in the second alarm array corresponding to the pre-configured item, b ij represents the jth element in the second alarm array corresponding to the host where the i-th fault Redis cluster is located, δ(b j ′,b ij ) is a comparison function (or indicator function), when (b j ′ =b ij ), returns 1, otherwise 0, n represents the total number of elements in the second alarm array, k iB A first ratio of the number of equal elements in the two arrays at all positions to the total number of elements in the second alarm array.

[0060] S3, performing a weighted summation of the first ratio and the second ratio using a preset weight coefficient to obtain a matching degree between the pre-configured item and the faulty Redis cluster;

[0061] S4, sorting the matching degrees corresponding to the pre-configuration items in the fault processing flow table from large to small. When the maximum matching degree is not less than a preset threshold, determining the fault type corresponding to the pre-configuration item corresponding to the maximum matching degree as the target fault type of the faulty Redis cluster.

[0062] Once the target fault type is determined, specific operations will be performed according to the processing strategy corresponding to the fault type in the fault processing flow table, such as adjusting Redis configuration parameters, restarting the service, and migrating to the backup cluster.

[0063] As an optional implementation, the following steps can be taken for optimization: when the maximum matching degree is less than a preset threshold, the faulty Redis cluster is processed according to a preset default processing strategy; a first prompt message is generated, and the first prompt message is used to prompt the target object to create a pre-configured item corresponding to the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

[0064] As an optional implementation, after executing the target processing strategy, a feedback and optimization process may also be executed, including: obtaining the first alarm array corresponding to the faulty Redis cluster again; generating a second prompt message when the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered. The second prompt message is used to prompt the target object to manually process the faulty Redis cluster and modify the processing strategy in the pre-configured item corresponding to the faulty Redis cluster in the fault processing flow table.

[0065] In an embodiment of the present application, by monitoring and collecting real-time alarm information from multiple Redis clusters associated with a business system and converting it into a first alarm array, the health status of the Redis cluster can be comprehensively and immediately understood. Based on the first alarm array, which Redis clusters are in a faulty state can be accurately identified, and the order of fault handling can be determined by calculating the fault handling priority index and combining it with the cluster's importance coefficient. This approach overcomes the lack of priority sorting in traditional fault handling, ensuring that Redis cluster faults in core businesses are handled first, thereby reducing business interruption time. After the faulty Redis cluster is determined, a second alarm array of the host on which it is located is further obtained. This introduces host-level fault information and improves the accuracy of fault location, because Redis cluster faults may be caused by hardware or system problems in the host. The fault handling flow table stores the mapping relationship between multiple combinations of the first alarm array and the second alarm array and multiple fault types, as well as the corresponding processing strategy for each fault type, realizing automated and intelligent fault handling. Operations and maintenance personnel do not need to manually analyze each alarm event, but can automatically match the fault type and execute the corresponding processing strategy, greatly shortening the time for fault response and resolution, thereby solving the technical problem of low efficiency of traditional manual handling of Redis cluster faults.

[0066] Example 2

[0067] According to the embodiments of the present application, the Redis cluster fault processing method is also provided, and the Redis cluster fault processing device is provided, as shown in the method and device. Figure 2 The Redis cluster fault processing device comprises at least an obtaining module 21, a determining module 22 and a processing module 23, wherein:

[0068] The obtaining module 21 is configured to obtain a plurality of first alarm arrays corresponding to a plurality of Redis clusters associated with a business system, wherein each first alarm array comprises a plurality of alarm event trigger states corresponding to a Redis cluster.

[0069] The determining module 22 is configured to determine a fault Redis cluster from the plurality of Redis clusters according to the first alarm array, and determine a processing priority of each fault Redis cluster.

[0070] The processing module 23 is configured to sequentially execute the following fault processing procedure on each fault Redis cluster according to the processing priority: obtaining a second alarm array corresponding to a host where the fault Redis cluster is located, wherein the second alarm array comprises a plurality of alarm event trigger states corresponding to the host; determining a target fault type matched with the first alarm array corresponding to the fault Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, and determining a target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault processing flow table stores a mapping relationship between a plurality of combinations of the first alarm array and the second alarm array and a plurality of fault types, and a processing strategy corresponding to each fault type.

[0071] The functions of each module of the Redis cluster fault processing device will be described in detail in combination with the specific implementation process.

[0072] The obtaining module obtains a plurality of first alarm arrays corresponding to a plurality of Redis clusters associated with a business system, wherein each first alarm array comprises a plurality of alarm event trigger states corresponding to a Redis cluster.

[0073] As an optional implementation, each element in the first alarm array corresponds to a trigger state of an alarm event of a Redis cluster, wherein the alarm event type of the Redis cluster comprises at least one of the following: cluster state exception, master node disconnection, space usage rate higher than a threshold, slave node disconnection, master-slave data synchronization failure, and node connection number higher than a threshold.

[0074] After obtaining the first alarm data, the determining module determines a fault Redis cluster from the plurality of Redis clusters according to the first alarm array, and determines a processing priority of each fault Redis cluster.

[0075] As an optional implementation, a faulty Redis cluster among multiple Redis clusters is determined based on the first alarm array. The process can take the following steps: for each Redis cluster, when the first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, the Redis cluster is determined to be a normal Redis cluster; when the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, the Redis cluster is determined to be a faulty Redis cluster.

[0076] As an optional implementation method, the processing priority of each faulty Redis cluster is determined. The process can take the following steps: for each faulty Redis cluster, determine the various alarm events triggered by the faulty Redis cluster based on the first alarm array corresponding to the faulty Redis cluster, and determine the preset alarm level corresponding to each triggered alarm event; sum the preset alarm levels corresponding to each triggered alarm event, and multiply the obtained sum by the preset importance coefficient corresponding to the faulty Redis cluster to obtain the priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; sort the various faulty Redis clusters from large to small according to the priority score to obtain the processing priority of each faulty Redis cluster.

[0077] After determining the priority, the processing module performs the following fault handling process for each faulty Redis cluster in turn based on the processing priority:

[0078] Obtain a second alarm array corresponding to the host where the faulty Redis cluster is located, where the second alarm array includes triggering statuses of multiple alarm events corresponding to the host;

[0079] Determine the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from the preconfigured fault processing flow table, determine the target processing strategy corresponding to the target fault type, and execute the target processing strategy. The fault processing flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

[0080] As an optional implementation, each element in the second alarm array corresponds to the triggering status of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

[0081] As an optional implementation, the fault handling flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a lifetime; the matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; the processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, and the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; the lifetime decays with the existence time of the pre-configuration item, and when the lifetime decays to 0, the pre-configuration item is automatically deleted, and when the pre-configuration item is matched by the faulty Redis cluster, the lifetime is automatically reset.

[0082] As an optional implementation, a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host is determined from a preconfigured fault processing flow table. The process can be carried out in the following steps: for each preconfigured item in the fault processing flow table, the first alarm array corresponding to the preconfigured item is compared with the elements at the same position in the first alarm array corresponding to the faulty Redis cluster one by one, and a first ratio of the number of identical elements to the total number of elements in the first alarm array is determined; the second alarm array corresponding to the preconfigured item is compared with the elements at the same position in the second alarm array corresponding to the host where the faulty Redis cluster is located, and a second ratio of the number of identical elements to the total number of elements in the second alarm array is determined; the first ratio and the second ratio are weightedly summed using a preset weight coefficient to obtain a matching degree between the preconfigured item and the faulty Redis cluster; the matching degrees corresponding to the preconfigured items in the fault processing flow table are sorted from large to small, and when the maximum matching degree is not less than a preset threshold, the fault type corresponding to the preconfigured item corresponding to the maximum matching degree is determined to be the target fault type of the faulty Redis cluster.

[0083] As an optional implementation, the following steps can be taken for optimization: when the maximum matching degree is less than a preset threshold, the faulty Redis cluster is processed according to a preset default processing strategy; a first prompt message is generated, and the first prompt message is used to prompt the target object to create a pre-configured item corresponding to the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

[0084] As an optional implementation, after executing the target processing strategy, the method further includes: obtaining the first alarm array corresponding to the faulty Redis cluster again; when the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, generating a second prompt message, the second prompt message being used to prompt the target object to manually process the faulty Redis cluster and modify the processing strategy in the pre-configured item corresponding to the faulty Redis cluster in the fault processing flow table.

[0085] It should be noted that the modules in the fault handling device for the Redis cluster in the embodiment of the present application correspond one-to-one to the implementation steps of the fault handling method for the Redis cluster in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0086] Example 3

[0087] According to an embodiment of the present application, a computer program product is further provided. The computer program product includes a computer program. When the computer program is executed by a processor, the fault handling method of the Redis cluster in Example 1 is implemented.

[0088] According to an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the Redis cluster fault handling method in Example 1 by running the computer program.

[0089] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the Redis cluster fault handling method in Example 1 is executed when the computer program is running.

[0090] According to an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the Redis cluster fault handling method in Example 1 through the computer program.

[0091] Specifically, when the computer program is running, the following steps are executed: obtaining multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each first alarm array includes the trigger status of multiple alarm events corresponding to a Redis cluster; determining the faulty Redis cluster among the multiple Redis clusters based on the first alarm array, and determining the processing priority of each faulty Redis cluster; executing the following fault processing process for each faulty Redis cluster in turn according to the processing priority: obtaining a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the trigger status of multiple alarm events corresponding to the host; determining a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, determining a target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault processing flow table stores a mapping relationship between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

[0092] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 The figure shows a hardware structure block diagram of an electronic device for implementing a fault handling method of a Redis cluster. Figure 3 As shown, the electronic device 30 may include one or more (illustrated as 302a, 302b, ..., 302n in the figure) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components than shown, or with Figure 3 Different configurations shown.

[0093] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the electronic device 30. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0094] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the fault handling method of the Redis cluster in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely located relative to the processor 302, and these remote memories may be connected to the electronic device 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0095] The transmission device 306 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the electronic device 30. In one embodiment, the transmission device 306 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 306 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0096] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 30 .

[0097] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0101] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0103] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A Redis cluster fault handling method, characterized in that: include: Acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with the business system, wherein each of the first alarm arrays includes triggering states of multiple alarm events corresponding to one Redis cluster; Determining a faulty Redis cluster among multiple Redis clusters according to the first alarm array, and determining a processing priority of each of the faulty Redis clusters; The following fault handling process is performed on each faulty Redis cluster in turn according to the processing priority: Obtain a second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes triggering states of multiple alarm events corresponding to the host; Determine a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, determine a target processing strategy corresponding to the target fault type, and execute the target processing strategy, wherein the fault processing flow table stores mapping relationships between combinations of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as a processing strategy corresponding to each fault type.

2. The method according to claim 1, characterized in that Each element in the first alarm array corresponds to the triggering status of an alarm event of the Redis cluster, wherein the alarm event type of the Redis cluster includes at least one of the following: abnormal cluster status, master node disconnection, space usage rate exceeding a threshold, slave node disconnection, master-slave data synchronization failure, and node connection number exceeding a threshold; Each element in the second alarm array corresponds to the triggering status of an alarm event of the host where the Redis cluster is located, wherein the alarm event type of the host where the Redis cluster is located includes at least one of the following: ping failure, CPU load higher than a threshold, insufficient memory, insufficient disk space, and network load higher than a threshold.

3. The method according to claim 1, characterized in that Determining a faulty Redis cluster among the multiple Redis clusters according to the first alarm array includes: For each Redis cluster, when the first alarm array corresponding to the Redis cluster indicates that multiple alarm events corresponding to the Redis cluster are not triggered, determining that the Redis cluster is a normal Redis cluster; When the first alarm array corresponding to the Redis cluster indicates that at least one alarm event corresponding to the Redis cluster is triggered, it is determined that the Redis cluster is a faulty Redis cluster.

4. The method according to claim 1, wherein Determine the processing priority of each of the faulty Redis clusters, including: For each faulty Redis cluster, determine each alarm event triggered by the faulty Redis cluster according to the first alarm array corresponding to the faulty Redis cluster, and determine the preset alarm level corresponding to each triggered alarm event; Sum the preset alarm levels corresponding to the triggered alarm events, and multiply the sum by the preset importance coefficient corresponding to the faulty Redis cluster to obtain a priority score corresponding to the faulty Redis cluster, wherein the preset importance coefficient is used to reflect the importance of the faulty Redis cluster to the business system and the impact of the fault of the faulty Redis cluster on the business system; The faulty Redis clusters are sorted from largest to smallest according to the priority scores to obtain a processing priority of each faulty Redis cluster.

5. The method according to claim 2, characterized in that The fault handling flow table includes multiple pre-configuration items, each of which includes: a matching domain, a processing strategy and a lifetime; The matching domain includes: a mapping relationship between a combination of a first alarm array and a second alarm array and a fault type; The processing strategy includes: a fault handling action and a fault handling method corresponding to the fault type, wherein the fault handling method includes at least one of the following: automatic execution, automatic execution after manual review; The survival time decays with the existence time of the pre-configuration item. When the survival time decays to 0, the pre-configuration item is automatically deleted. When the pre-configuration item is matched by a faulty Redis cluster, the survival time is automatically reset.

6. The method according to claim 5, characterized in that Determining a target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table includes: For each pre-configured item in the fault handling flow table, compare the first alarm array corresponding to the pre-configured item with the elements at the same position in the first alarm array corresponding to the faulty Redis cluster one by one, and determine a first ratio of the number of identical elements to the total number of elements in the first alarm array; Compare the elements at the same position in the second alarm array corresponding to the pre-configured item and the second alarm array corresponding to the host where the faulty Redis cluster is located one by one, and determine a second ratio of the number of the same elements to the total number of elements in the second alarm array; Performing a weighted summation of the first ratio and the second ratio using a preset weight coefficient to obtain a matching degree between the pre-configured item and the faulty Redis cluster; The matching degrees corresponding to the pre-configuration items in the fault processing flow table are sorted from large to small. When the maximum matching degree is not less than a preset threshold, the fault type corresponding to the pre-configuration item corresponding to the maximum matching degree is determined to be the target fault type of the faulty Redis cluster.

7. The method according to claim 6, characterized in that The method further comprises: When the maximum matching degree is less than the preset threshold, the faulty Redis cluster is processed according to a preset default processing strategy; Generate first prompt information, which is used to prompt the target object to create pre-configured items corresponding to a first alarm array corresponding to the faulty Redis cluster and a second alarm array corresponding to the host where the faulty Redis cluster is located in the fault processing flow table.

8. The method according to claim 5, characterized in that After executing the target processing strategy, the method further includes: Obtain the first alarm array corresponding to the faulty Redis cluster again; When the first alarm array indicates that at least one alarm event corresponding to the faulty Redis cluster is still triggered, second prompt information is generated. The second prompt information is used to prompt the target object to manually handle the faulty Redis cluster and modify the processing strategy in the pre-configured item corresponding to the faulty Redis cluster in the fault processing flow table.

9. A fault handling device for a Redis cluster, characterized in that: include: An acquisition module, configured to acquire multiple first alarm arrays corresponding to multiple Redis clusters associated with a business system, wherein each of the first alarm arrays includes triggering states of multiple alarm events corresponding to one Redis cluster; a determination module, configured to determine a faulty Redis cluster among multiple Redis clusters based on the first alarm array, and determine a processing priority for each of the faulty Redis clusters; A processing module is used to execute the following fault processing process for each of the faulty Redis clusters in turn according to the processing priority: obtaining the second alarm array corresponding to the host where the faulty Redis cluster is located, wherein the second alarm array includes the trigger status of multiple alarm events corresponding to the host; determining the target fault type that matches the first alarm array corresponding to the faulty Redis cluster and the second alarm array corresponding to the host from a preconfigured fault processing flow table, and determining the target processing strategy corresponding to the target fault type, and executing the target processing strategy, wherein the fault processing flow table stores the mapping relationship between the combination of multiple first alarm arrays and second alarm arrays and multiple fault types, as well as the processing strategy corresponding to each fault type.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the Redis cluster fault handling method according to any one of claims 1 to 8 through the computer program.

Images