Big data cluster fault analysis method, device, equipment, medium and product

Abstract

The present application discloses a method, device, equipment, medium and product for analyzing a big data cluster fault, wherein the method for analyzing a big data cluster fault includes: obtaining a first directed acyclic graph composed of a plurality of first analysis models; determining a first topological order of the plurality of first analysis models according to the first directed acyclic graph; obtaining a plurality of first analysis tasks generated by the plurality of first analysis models, each first analysis model generating a first analysis task; executing the first analysis task generated by each first analysis model according to the first topological order, and obtaining a first analysis result corresponding to each first analysis task; determining a fault analysis result of an analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task. According to the embodiments of the present application, faults can be accurately predicted and located, and the accuracy of fault analysis can be improved.

Description

Technical Field

[0001] The present application belongs to the field of computer information technology, and in particular, relates to a big data cluster fault analysis method, device, equipment, medium and product. Background Art

[0002] Usually, big data clusters are deployed on multiple server nodes, each of which plays a different role. The cluster includes multiple components, each of which contains multiple services, which are generally distributed on different host nodes. The physical dispersion of signals brings great difficulties to the operation and maintenance of big data clusters. In addition, the nonlinear relationship between components and services determines that it is impossible to analyze faults through one or two component services alone.

[0003] When analyzing faults in a big data cluster, the existing method is to monitor the indicators of each big data component separately and display them in a centralized manner through a visualization platform. Usually, the content that can be monitored and analyzed is the indicator status application program interface list of each major data component. This analysis mode mainly relies on the indicators provided by the component itself for analysis, and the prediction and location of faults rely on the experience of operation and maintenance personnel, resulting in low accuracy of fault analysis. Summary of the invention

[0004] The embodiments of the present application provide a big data cluster fault analysis method, device, equipment, medium and product, which can improve the accuracy of fault analysis.

[0005] In a first aspect, an embodiment of the present application provides a big data cluster fault analysis method, the method comprising:

[0006] Obtaining a first directed acyclic graph consisting of a plurality of first analysis models, wherein each node in the first directed acyclic graph corresponds to a first analysis model, and each first analysis model is used to analyze indicator data and / or log data of an analysis object in the cluster;

[0007] Determining a first topological order of the plurality of first analysis models according to the first directed acyclic graph;

[0008] Acquire a plurality of first analysis tasks generated by a plurality of first analysis models, each first analysis model generating a first analysis task;

[0009] Execute the first analysis task generated by each first analysis model according to the first topological order to obtain a first analysis result corresponding to each first analysis task;

[0010] According to the first analysis result corresponding to each first analysis task, the fault analysis result of the analysis object corresponding to each first analysis task is determined.

[0011] In a second aspect, an embodiment of the present application provides a big data cluster fault analysis device, the device comprising:

[0012] A first acquisition module, used to acquire a first directed acyclic graph composed of multiple first analysis models, each node in the first directed acyclic graph corresponds to a first analysis model, and each first analysis model is used to analyze indicator data and / or log data of an analysis object in the cluster;

[0013] A first determining module, configured to determine a first topological order of a plurality of first analysis models according to a first directed acyclic graph;

[0014] A second acquisition module is used to acquire a plurality of first analysis tasks generated by a plurality of first analysis models, each first analysis model generates a first analysis task;

[0015] An execution module, configured to execute the first analysis task generated by each first analysis model according to the first topological order, and obtain a first analysis result corresponding to each first analysis task;

[0016] The second determination module is used to determine the fault analysis result of the analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0018] When the processor executes the computer program instructions, the method according to the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method described in the first aspect.

[0021] The big data cluster fault analysis method, device, equipment, medium and product of the embodiments of the present application obtain a first directed acyclic graph composed of multiple first analysis models, wherein each node in the first directed acyclic graph corresponds to a first analysis model, and the multiple first analysis models generate multiple first analysis tasks. The first analysis model is used to analyze the indicator data and / or log data of an analysis object in the cluster, and the indicator data and log data are simultaneously acquired and stored, thereby avoiding multi-channel transmission of data, shortening the data transmission link, and facilitating unified management of data. The first topological order of the multiple first analysis models is determined according to the first directed acyclic graph, and the multiple first analysis tasks are executed according to the determined first topological order, so that the first analysis result corresponding to the first analysis task can be obtained, and the fault analysis result of the analysis object corresponding to the first analysis task can be determined according to the first analysis result. In this way, through the combination of multiple analysis models and cluster topology, a more comprehensive and systematic analysis of the state of the analysis object can be performed, and then the fault can be accurately predicted and located, thereby improving the efficiency of fault analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 It is a flowchart of a big data cluster fault analysis method provided by an embodiment of the present application;

[0024] Figure 2 It is a structural diagram of cluster topology perception in a big data cluster fault analysis method provided by an embodiment of the present application;

[0025] Figure 3 It is a structural diagram of the analysis of indicator data and log data in a big data cluster fault analysis method provided by an embodiment of the present application;

[0026] Figure 4 It is a structural diagram of a task execution call chain in a big data cluster fault analysis method provided by an embodiment of the present application;

[0027] Figure 5 This is a structural diagram of data transmission and storage in a big data cluster fault analysis method provided by an embodiment of the present application;

[0028] Figure 6 It is a structural diagram of a big data cluster fault analysis device provided by an embodiment of the present application;

[0029] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0032] In order to solve the problems of the prior art, the embodiments of the present application provide a big data cluster fault analysis method, device, equipment, medium and product. The big data cluster fault analysis method provided by the embodiments of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0033] Figure 1 FIG. 1 is a flow chart of a method for analyzing a big data cluster failure provided by an embodiment of the present application. Figure 1 As shown, the big data cluster fault analysis method provided in the embodiment of the present application may include steps S110-S150, wherein:

[0034] S110, obtaining a first directed acyclic graph consisting of a plurality of first analysis models;

[0035] S120, determining a first topological order of a plurality of first analysis models according to the first directed acyclic graph;

[0036] S130, obtaining a plurality of first analysis tasks generated by a plurality of first analysis models, wherein each first analysis model generates a first analysis task;

[0037] S140, executing each first analysis task generated by the first analysis model according to the first topological order, and obtaining a first analysis result corresponding to each first analysis task;

[0038] S150: Determine the fault analysis result of the analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task.

[0039] Therefore, by obtaining a first directed acyclic graph consisting of multiple first analysis models, wherein each node in the first directed acyclic graph corresponds to a first analysis model, multiple first analysis models generate multiple first analysis tasks, the first analysis model is used to analyze the indicator data and / or log data of an analysis object in the cluster, and the indicator data and log data are simultaneously acquired and stored, thereby avoiding multi-path transmission of data, shortening the data transmission link, and facilitating unified management of data. The first topological order of the multiple first analysis models is determined according to the first directed acyclic graph, and multiple first analysis tasks are executed according to the determined first topological order, so that the first analysis result corresponding to the first analysis task can be obtained, and the fault analysis result of the analysis object corresponding to the first analysis task can be determined according to the first analysis result. In this way, through the combination of multiple analysis models and cluster topology, a more comprehensive and systematic analysis of the state of the analysis object can be performed, and then the fault can be accurately predicted and located, thereby improving the efficiency of fault analysis.

[0040] The specific implementation methods of the above steps are introduced below.

[0041] In one embodiment of the present application, in S110, the first analysis model can be an operation status analysis model, the first directed acyclic graph can represent the connection relationship between multiple first analysis models, the first directed acyclic graph can be pre-set, and each first analysis model can analyze the analysis object in the cluster, wherein the analysis object is a resource type in the cluster topology, that is, one of the four types of cluster, service, component, and host and its specific identification name. Each first analysis model corresponds to a different analysis object, and each first analysis model is constructed in three parts, including analysis objects, indicator judgment rules and thresholds, and log search rules and keywords.

[0042] The big data cluster fault analysis system of the present application includes a data acquisition and processing module and method, a real-time data transmission and input library module and method, a cluster topology perception module and method, an indicator and log analysis module and method, and a monitoring alarm and fault analysis large screen module and method.

[0043] For example, Figure 2 As shown, Figure 2It is a structural diagram of cluster topology perception in a big data cluster fault analysis method provided by an embodiment of the present application. The big data cluster fault analysis system of the present application provides a cluster topology perception module and method. The platform service detector collects information such as adding nodes, reducing nodes, adding services, reducing services, adding components, reducing components, and service operation status changes in real time to achieve real-time perception of changes in big data cluster node deployment information and node service operation status. Furthermore, the data will be saved to the Configuration Management Database (CMDB) database.

[0044] In each federated cluster or multi-cluster, there will be multiple nodes and management nodes. The platform service detector scans multiple cluster nodes of big data, obtains the installation and deployment status of node component services and service operation status, etc., places the detection results in a specific resource path, and provides a platform service automatic push device to synchronously push the detection results to the CMDB library.

[0045] The platform service detector scans the detection point and stores the latest cluster service distribution and status to a specific resource path in a linear relationship of "cluster-component-service-host-address-service status-timestamp". The platform service automatic push device intercepts the heartbeat communication duration between the management node and the storage node in the heartbeat mechanism. If the communication duration threshold is exceeded, the alarm device is triggered and the alarm information is pushed to the monitoring alarm module, notifying the indicator and log analysis module to conduct active fault analysis on the indicators and logs of the analysis object with abnormal signals.

[0046] Through the above embodiments, the cluster topology awareness module avoids the cumbersome deployment of traditional collection. There is no need to deploy the client in each cluster slave node, and it only needs to be deployed in the management node. In addition, the cluster topology awareness module supports the monitoring of the distribution and status of component services of a single cluster or a federated cluster. The cluster topology awareness module can also detect monitoring items and support adaptation and increase, and the configuration can be changed to adjust cluster monitoring.

[0047] In one embodiment of the present application, in S120, the first topological order of multiple first analysis models can be determined according to the first directed acyclic graph mentioned above. In order to ensure the correctness of the fault analysis order and dependency relationship, the execution order of the first analysis task will be carried out according to the multiple first topological orders of the first analysis model in the directed acyclic graph.

[0048] In one embodiment of the present application, in S130, if Figure 3 As shown, Figure 3It is a structural diagram of the analysis of indicator data and log data in a big data cluster fault analysis method provided by an embodiment of the present application. The knowledge base is composed of multiple directed acyclic graphs composed of multiple analysis models input by operation and maintenance experts, and multiple first analysis tasks generated by multiple first analysis models are obtained, and multiple first analysis tasks are constructed. In order to improve performance, multiple first analysis tasks need to be executed concurrently. Therefore, in the process of multiple first topological sequences, multiple execution stages will be divided. In each stage, all first analysis models with in-degree 0 are selected and the first analysis tasks are constructed and put into the queue. After all the first analysis tasks in the queue are executed, the first analysis model nodes that have just been executed are eliminated from the temporarily obtained directed acyclic graph, and then the next round of task construction and execution is carried out.

[0049] In one embodiment of the present application, in S140, executing the first analysis task generated by each first analysis model according to the first topological order to obtain the first analysis result corresponding to each first analysis task includes the following two steps:

[0050] Step 1: For each second task, obtain indicator data of the first analysis object of the second model within a preset time period, the second task is any one of the multiple first analysis tasks, and the second model is a model for generating the second task.

[0051] Step 2: According to the indicator judgment rule and threshold of the second model, the indicator data is judged to obtain the first sub-analysis result, which is used to indicate the health level of the indicator data.

[0052] In the above, the first analysis model includes the identification of the first analysis object, the indicator judgment rules and thresholds of the first analysis object, and the information of the indicator judgment rules and thresholds includes the indicator identification name, and for three health levels: normal, risk, and abnormal, there are three sets of judgment rules and corresponding thresholds written in a specific domain language.

[0053] Exemplarily, the big data cluster fault analysis system of the present application provides an indicator and log analysis module and method, which features include a knowledge base sub-module and construction method and an analysis machine sub-module and method, to achieve big data component operation status analysis and rapid fault location.

[0054] Obtain indicator data of the first analysis object of the second model within a preset time period, wherein the time period can be custom set, the second model is a model for generating a second task, and the second task is any one of the multiple first analysis tasks. According to the indicator judgment rule and threshold of the second model, the indicator data of the first analysis object is judged to obtain a first sub-analysis result. The first sub-analysis result can indicate the health level of the indicator data, and the health level may include normal, risk, and abnormal. For example, if the component indicator value of the first analysis object is greater than the set first threshold, the first analysis result displays abnormality; if the component indicator value of the first analysis object is less than the set second threshold, the first analysis result displays normal; if the component indicator value of the first analysis object is between the set first threshold and the second threshold, the first analysis result displays risk.

[0055] In one embodiment of the present application, in S140, the first analysis task generated by each first analysis model is executed according to the first topological order to obtain the first analysis result corresponding to each first analysis task, and the following three steps are also included:

[0056] Step 3: Search the logs stored in the database within a preset time period according to the log search rule and search keywords of the first analysis object of the second model to obtain first log data, and the database is used to store the log data of the analysis object;

[0057] Step 4: judge the first log data to obtain the second sub-analysis result, where the second sub-analysis result is used to indicate the health of the first log data;

[0058] Step 5: The one with lower health level between the first sub-analysis result and the second sub-analysis result is used as the first analysis result corresponding to the second task.

[0059] In the above, the first analysis model also includes log search rules and search keywords for the first analysis object. The log search rules and keywords include log identification names, as well as search rules and keywords for three health levels. The search rules are time window size and keyword matching patterns.

[0060] Exemplarily, the first log data is obtained according to the log search rules and search keywords of the first analysis object of the second model within a preset time period in the database, wherein the time period can be custom set, the second model is a model for generating the second task, and the second task is any one of the multiple first analysis tasks. The database is used to store the log data of the analysis object, and the first log data of the first analysis object is judged to obtain the second sub-analysis result. The first sub-analysis result can represent the health level of the first log data, and the health level decreases in order of normal, risk, and abnormal. The sub-analysis result of a single analysis task takes the lower of the indicator and the log health result. Each analysis task will generate an analysis task health result based on the dependent analysis model connected to the input edge of its analysis model in the directed acyclic graph, and take the lower of the health results of its own analysis task based on the analysis task health result, and output it to the result library as the final analysis result. The first sub-analysis result and the second sub-analysis result are compared, and the one with the lower health level is compared as the first analysis result corresponding to the second task.

[0061] Through the above embodiments, a topic model is built for each log indicator, which is convenient for unified operation and maintenance management and analysis, and real-time incremental data acquisition is achieved, and transmission is achieved in seconds, ensuring the timeliness of data. The real-time data warehouse responds quickly, achieving a response in seconds, without delay under large data volumes, and the analysis model customizes the judgment rule machine threshold, which can flexibly adapt the analysis methods of resource entities at different levels.

[0062] In one embodiment of the present application, in S150, determining the fault analysis result of the analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task includes the following two steps:

[0063] Step 6: If the first analysis model corresponding to the first task has no incoming edge in the first directed acyclic graph, the fault analysis result of the analysis object corresponding to the first task is determined as the first analysis result corresponding to the first task, wherein the first task is any one of the multiple first analysis tasks;

[0064] Step 7: If the first analysis model corresponding to the first task has an incoming edge in the first directed acyclic graph, the fault analysis result of the analysis object corresponding to the first task is determined based on the first analysis result corresponding to the first task and the first analysis result of the first analysis task corresponding to the first model, wherein the incoming edge of the first analysis model corresponding to the first task in the first directed acyclic graph is the first model.

[0065] Exemplarily, multiple first analysis models are connected by directed edges in the analysis order, that is, an analysis model A will point to another analysis model B that depends on the analysis result of analysis model A. There are two types of analysis models, namely, operation status class and fault location class. These two types of analysis models will be run in different scheduling modes in the analysis machine. In the same directed acyclic graph, there will only be one type of analysis model.

[0066] The analysis machine is a program that performs operation status analysis and fault location actions. It will perform big data component operation analysis and rapid fault location based on the cluster topology perception information and the analysis model in the knowledge base. During initialization, the cluster, service, component, and host relationships in the cluster topology perception module will be loaded first, and these entity relationships will be instantiated in the analysis machine memory as analysis objects. After initialization, the analysis machine will differentiate into two sub-analyzers with different working modes: a clock-driven sub-analyzer and an event-driven sub-analyzer.

[0067] The sub-analysis machine in clock-driven mode is used to implement the function of analyzing and outputting the running status of big data components. It will periodically obtain the running status class analysis model with relevant analysis objects and its directed acyclic graph from the knowledge base according to the instantiated cluster topology, construct an independent analysis task execution queue for each cluster, and then execute the analysis tasks of multiple execution queues in parallel.

[0068] In the above execution queue of each cluster, each analysis model will generate an analysis task. The analysis machine injects the actual attributes of the cluster into the analysis object of the analysis model, determines the specific analysis object of the analysis model, and uses this as one of the conditions for querying indicator data and log data. According to the indicator judgment rules and the indicator identification name, time and analysis object of the threshold, the required indicator data is taken out from the indicator library to determine the health of the indicator data of the analysis model. At the same time, according to the log search rules and keywords, the log identification name, time window and analysis object are used as conditions to match from the log library to determine the health of the log data of the analysis model.

[0069] When the first analysis model corresponding to the first task has no incoming edge in the first directed acyclic graph, the fault analysis result is the first analysis result corresponding to the first task; when the first analysis model corresponding to the first task has an incoming edge in the first directed acyclic graph, the fault analysis result is determined not only by the first analysis result corresponding to the first task, but also by the first analysis result of the first analysis task corresponding to the first model, and the first model is the incoming edge of the first analysis model corresponding to the first task in the first directed acyclic graph.

[0070] Through the above embodiments, by combining the analysis model and its relationship diagram in the knowledge base with the cluster topology, a more comprehensive system analysis of the state of an analysis object can be performed, and the states of the components and hosts on which it depends can be perceived.

[0071] In one embodiment of the present application, after S150, the following steps are further included:

[0072] Step 1: obtaining a second directed acyclic graph consisting of a plurality of second analysis models according to a preset fault event, wherein each node in the second directed acyclic graph corresponds to a second analysis model, and each second analysis model is used to obtain a fault analysis result of an analysis object in the cluster;

[0073] Step 2: determining a second topological order of the plurality of second analysis models according to the second directed acyclic graph;

[0074] Step 3: a fourth acquisition module, used to acquire multiple second analysis tasks generated by multiple second analysis models, where each second analysis model generates one second analysis task;

[0075] Step 4: Execute the second analysis task generated by each second analysis model according to the second topological order to obtain the second analysis result corresponding to each second analysis task;

[0076] Step 5: Determine the analysis object of the fault event according to the second analysis result corresponding to each second analysis task.

[0077] In the above, the second analysis model may be a fault location analysis model, and the second directed acyclic graph may be pre-set.

[0078] Exemplarily, a second directed acyclic graph consisting of multiple second models is obtained according to a preset fault event, and a rapid fault location function is implemented through a sub-analysis machine in an event-driven mode. The second topological order of multiple second analysis models is determined according to the second directed acyclic graph, and the fault location analysis model with relevant fault event labels and analysis objects and the directed acyclic graph in which it is located are obtained in the knowledge base through the fault event information received by the monitored distributed message queue and other interface services, and an independent second analysis task execution queue is constructed for each cluster, and the second analysis tasks of multiple execution queues are executed in parallel to obtain the second analysis results corresponding to the second analysis tasks, thereby determining the analysis object where the fault event occurred. The construction and execution process of the subsequent execution queue are the same as those of the aforementioned clock-driven sub-analysis machine.

[0079] In one embodiment of the present application, before S110, the following steps are also included:

[0080] Step 6: Use different indicator collection methods to collect indicator data from different indicator data sources through the indicator collector;

[0081] Step 7: Report the indicator data collected by the indicator collector through the indicator receiver and write it into the distributed message queue;

[0082] Step 8: Store the indicator data in the message queue into the database.

[0083] Exemplarily, each first analysis model analyzes the indicator data and / or log data of an analysis object in the cluster. It should be noted that the indicator data and log data of the analysis object may be obtained at the same time, or any one of them may be obtained.

[0084] like Figure 4 As shown, Figure 4 It is a structural diagram of the task execution call chain in the big data cluster fault analysis method provided by an embodiment of the present application. An indicator collection submodule in a data acquisition and processing module is provided, which is used to write the indicator data of the analysis object into a distributed message queue. Its characteristics include indicator collection and indicator reception. The indicator collector uses different indicator collection working modes according to different indicator data sources. For example, big data components usually provide indicator output interfaces of Hypertext Transfer Protocol (Http) and Remote Method Invocation (Rmi). Then this type of indicator collector runs the active collection process of the corresponding protocol. For the program to be monitored without an indicator interface, a matching indicator collector is used to run a customized active collection process, and the source layer indicator data is generated and sent to the indicator receiver. The collection scheduling plan can be dynamically adjusted according to business needs and cluster load.

[0085] The indicator receiver runs in a distributed manner, receives data reported by the indicator collector in real time and writes it into a distributed message queue. It ensures high availability and load balancing through distributed coordination services. Even when some indicator receivers fail, the indicator collector will switch to send data to other available indicator receivers. For complex network environments, cascade forwarding can be performed between indicator receivers.

[0086] In one embodiment of the present application, before S110, the following steps are also included:

[0087] Step 9: Use the log collector to collect the newly added log text content of the analysis object in real time;

[0088] Step 10: Parse the log text content according to the log type of the log text content through the log processor, extract the structured fields, and write the structured fields into the distributed message queue;

[0089] Step 11: Store the structured fields in the message queue into the database.

[0090] Exemplarily, a log collection submodule in a data acquisition and processing module is also provided, which is used to monitor the log data of the analysis object and write it into a distributed message queue after parsing. Its features include log collection and log reception. The log collector runs on a bare metal device or container that needs to collect logs. It can monitor the changes of the specified text file, collect the newly added log text content in real time and send it to the log receiver. The log receiver runs in a distributed manner, receives the data reported by the log collector in real time, and performs different text parsing according to the log type, extracts structured fields, that is, log data, and then sends it to the distributed message queue. Like the indicator receiver, the log receiver ensures high availability and load balancing through distributed coordination services, and can perform data transparent transmission in a complex network through cascade forwarding.

[0091] Through the above embodiments, the indicator collection and log collection logic are actively performed by the collector, and the strategy can be adjusted systematically. The collector discovers the receiver through the distributed coordination service, achieving high availability and load balancing. The receiver supports cascade forwarding and does not require additional configuration of agents in complex networks.

[0092] like Figure 5 As shown, Figure 5 It is a structural diagram of data transmission and storage in a big data cluster fault analysis method provided by an embodiment of the present application.

[0093] The big data cluster fault system of the present application provides a data transmission and warehousing module and method, which features include real-time synchronous distributed publish-subscribe message system (Kafka Cluster, Kafka) and hypertext transfer protocol (Hyper texttransport protocol, Http) indicator data stream and log data stream, creating topics corresponding to various indicator data and log data in Kafka, using a real-time stream processing engine to filter, clean, parse, and format the source data, and output it to the corresponding topic of Kafka according to category, and connecting to Kafka's data consumption module to read the indicators and logs in the message queue in real time.

[0094] The data consumption module uses the Change Data Capture (CDC) specification to extract incremental indicators and log data to the source data layer in real time, build a real-time data warehouse theme model layer, classify and summarize indicators and logs from different hosts, and build triggers in the real-time data warehouse to monitor changes in source layer data. Once new data is added, the stored procedure is used to convert the source layer data and load it into the theme model layer.

[0095] In addition, the big data cluster fault analysis system of the present application also provides a monitoring alarm and fault analysis large-screen module and analysis method, which are used to display the operating status and fault analysis of the big data components managed and monitored by this system. All resource entities (clusters, services, components, hosts) and their relationships in cluster topology perception will be displayed on the first page of the large-screen module, and displayed in a visual tree structure on the large-screen page. At the same time, the operating status of these resource entities as analysis objects will be displayed in real time from the result library. All fault events are displayed on the second page of the large-screen module. For each fault event, all associated analysis results generated by the event-driven fault rapid location analysis function can be viewed from its details page.

[0096] It should be noted that the application scenarios described in the above-mentioned embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0097] Based on the same inventive concept, the present application also provides a big data cluster fault analysis device. Figure 6 Provide detailed explanation.

[0098] Figure 6 Schematic diagram of the structure of a big data cluster fault analysis device provided by an embodiment of the present application. Figure 6 As shown, the big data cluster fault analysis device 600 includes:

[0099] A first acquisition module 601 is used to acquire a first directed acyclic graph consisting of a plurality of first analysis models, wherein each node in the first directed acyclic graph corresponds to a first analysis model, and each first analysis model is used to analyze indicator data and / or log data of an analysis object in the cluster;

[0100] A first determining module 602 is used to determine a first topological order of a plurality of first analysis models according to a first directed acyclic graph;

[0101] A second acquisition module 603 is used to acquire a plurality of first analysis tasks generated by a plurality of first analysis models, where each first analysis model generates a first analysis task;

[0102] An execution module 604 is used to execute the first analysis task generated by each first analysis model according to the first topological order to obtain a first analysis result corresponding to each first analysis task;

[0103] The second determination module 605 is used to determine the fault analysis result of the analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task.

[0104] In one embodiment of the present application, the second determining module 605 includes:

[0105] A first determination submodule, configured to determine the fault analysis result of the analysis object corresponding to the first task as the first analysis result corresponding to the first task if the first analysis model corresponding to the first task has no incoming edge in the first directed acyclic graph, wherein the first task is any one of the plurality of first analysis tasks;

[0106] The second determination submodule is used to determine the fault analysis result of the analysis object corresponding to the first task according to the first analysis result corresponding to the first task and the first analysis result of the first analysis task corresponding to the first model if the first analysis model corresponding to the first task has an incoming edge in the first directed acyclic graph, wherein the incoming edge of the first analysis model corresponding to the first task in the first directed acyclic graph is the first model.

[0107] In one embodiment of the present application, each first analysis model includes an identifier of a first analysis object, an indicator judgment rule and a threshold value of the first analysis object, and the execution module 604 includes:

[0108] A first acquisition submodule is used to acquire, for each second task, indicator data of a first analysis object of a second model within a preset time period, where the second task is any one of the plurality of first analysis tasks, and the second model is a model for generating the second task;

[0109] The first judgment submodule is used to judge the indicator data according to the indicator judgment rule and threshold of the second model to obtain a first sub-analysis result, and the first sub-analysis result is used to indicate the health of the indicator data.

[0110] In one embodiment of the present application, each first analysis model further includes a log search rule and a search keyword of the first analysis object, and the execution module 604 includes:

[0111] A search submodule, for searching the logs stored in a preset time period in a database according to the log search rule and search keywords of the first analysis object of the second model to obtain first log data, where the database is used to store the log data of the analysis object;

[0112] A second judgment submodule is used to judge the first log data to obtain a second sub-analysis result, where the second sub-analysis result is used to indicate the health of the first log data;

[0113] As a submodule, it is used to use the one with a lower health level between the first sub-analysis result and the second sub-analysis result as the first analysis result corresponding to the second task.

[0114] In one embodiment of the present application, the device 600 further includes:

[0115] A third acquisition module is used to acquire a second directed acyclic graph consisting of multiple second analysis models according to a preset fault event, each node in the second directed acyclic graph corresponds to a second analysis model, and each second analysis model is used to obtain a fault analysis result of an analysis object in the cluster;

[0116] A third determining module, configured to determine a second topological order of the plurality of second analysis models according to the second directed acyclic graph;

[0117] A fourth acquisition module, used to acquire a plurality of second analysis tasks generated by a plurality of second analysis models, each second analysis model generating a second analysis task;

[0118] A generating module, used for executing the second analysis task generated by each second analysis model according to the second topological order, and obtaining a second analysis result corresponding to each second analysis task;

[0119] The fourth determination module is used to determine the analysis object of the fault event according to the second analysis result corresponding to each second analysis task.

[0120] In one embodiment of the present application, the device 600 further includes:

[0121] The collection module is used to collect indicator data from different indicator data sources using different indicator collection methods through the indicator collector;

[0122] The reporting module is used to report the indicator data collected by the indicator collector through the indicator receiver and write it into the distributed message queue;

[0123] The first storage module is used to store the indicator data in the message queue into the database.

[0124] In one embodiment of the present application, the device 600 further includes:

[0125] The collection module is used to collect the newly added log text content of the analysis object in real time through the log collector;

[0126] The parsing module is used to parse the log text content according to the log type of the log text content through the log processor, extract the structured fields, and write the structured fields into the distributed message queue;

[0127] The second storage module is used to store the structured fields in the message queue into a database.

[0128] Therefore, by obtaining a first directed acyclic graph composed of multiple first analysis models, wherein each node in the first directed acyclic graph corresponds to a first analysis model, multiple first analysis models generate multiple first analysis tasks, the first analysis model is used to analyze the indicator data and / or log data of an analysis object in the cluster, and the indicator data and log data are simultaneously acquired and stored, thereby avoiding multi-channel transmission of data, shortening the data transmission link, and facilitating unified management of data. The first topological order of the multiple first analysis models is determined according to the first directed acyclic graph, and multiple first analysis tasks are executed according to the determined first topological order, so that the first analysis result corresponding to the first analysis task can be obtained, and the fault analysis result of the analysis object corresponding to the first analysis task can be determined according to the first analysis result. In this way, through the combination of multiple analysis models and cluster topology, a more comprehensive and systematic analysis of the state of the analysis object can be performed, and then the fault can be accurately predicted and located, thereby improving the efficiency of fault analysis.

[0129] The big data cluster fault analysis device 600 provided in the embodiment of the present application can implement each process implemented in the aforementioned big data cluster fault analysis method embodiment, and will not be described again here to avoid repetition.

[0130] Figure 7 A schematic diagram of the hardware structure of a big data cluster fault analysis method provided in an embodiment of the present application is shown.

[0131] The electronic device may include a processor 701 and a memory 702 storing computer program instructions.

[0132] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0133] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 702 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory.

[0134] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect or the second aspect of the present disclosure.

[0135] The processor 701 implements any one of the big data cluster fault analysis methods in the above embodiments by reading and executing computer program instructions stored in the memory 702 .

[0136] In one example, the electronic device may further include a communication interface 703 and a bus 710. Figure 7 As shown, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 710 and communicate with each other.

[0137] The communication interface 703 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0138] Bus 710 includes hardware, software or both, and the parts of big data cluster fault analysis method or verification equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 710 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0139] In addition, in combination with the big data cluster fault analysis method in the above embodiment, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the big data cluster fault analysis methods in the above embodiment is implemented.

[0140] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0141] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0142] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0143] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0144] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A big data cluster fault analysis method, characterized in that: The method comprises: Obtaining a first directed acyclic graph consisting of a plurality of first analysis models, wherein each node in the first directed acyclic graph corresponds to a first analysis model, and each first analysis model is used to analyze indicator data and / or log data of an analysis object in the cluster; Determining a first topological order of the plurality of first analysis models according to the first directed acyclic graph; Acquire a plurality of first analysis tasks generated by the plurality of first analysis models, each first analysis model generating a first analysis task; Execute the first analysis task generated by each first analysis model according to the first topological order to obtain a first analysis result corresponding to each first analysis task; According to the first analysis result corresponding to each first analysis task, the fault analysis result of the analysis object corresponding to each first analysis task is determined.

2. The method according to claim 1, characterized in that The determining, according to the first analysis result corresponding to each first analysis task, the fault analysis result of the analysis object corresponding to each first analysis task includes: If the first analysis model corresponding to the first task has no incoming edge in the first directed acyclic graph, determining the first analysis result corresponding to the first task as the fault analysis result of the analysis object corresponding to the first task, wherein the first task is any one of the multiple first analysis tasks; If the first analysis model corresponding to the first task has an incoming edge in the first directed acyclic graph, the fault analysis result of the analysis object corresponding to the first task is determined based on the first analysis result corresponding to the first task and the first analysis result of the first analysis task corresponding to the first model, wherein the incoming edge of the first analysis model corresponding to the first task in the first directed acyclic graph is the first model.

3. The method according to claim 1, characterized in that Each first analysis model includes an identifier of a first analysis object, an indicator judgment rule and a threshold value of the first analysis object; The step of sequentially executing each first analysis task generated by the first analysis model according to the first topology to obtain a first analysis result corresponding to each first analysis task includes: For each second task, obtaining indicator data of a first analysis object of a second model within a preset time period, wherein the second task is any one of the plurality of first analysis tasks, and the second model is a model for generating the second task; The indicator data is judged according to the indicator judgment rule and threshold of the second model to obtain a first sub-analysis result, where the first sub-analysis result is used to indicate the health level of the indicator data.

4. The method according to claim 3, characterized in that Each first analysis model also includes log search rules and search keywords for the first analysis object; The method further includes: executing the first analysis task generated by each first analysis model according to the first topological order to obtain the first analysis result corresponding to each first analysis task; According to the log search rule and search keywords of the first analysis object of the second model, the logs stored in the preset time period in the database are searched to obtain first log data, and the database is used to store the log data of the analysis object; Judging the first log data to obtain a second sub-analysis result, where the second sub-analysis result is used to indicate the health of the first log data; The one with the lower health level between the first sub-analysis result and the second sub-analysis result is used as the first analysis result corresponding to the second task.

5. The method according to claim 1, characterized in that After determining the fault analysis result of the analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task, the method further includes: Acquire a second directed acyclic graph consisting of a plurality of second analysis models according to a preset fault event, wherein each node in the second directed acyclic graph corresponds to a second analysis model, and each second analysis model is used to obtain a fault analysis result of an analysis object in the cluster; determining a second topological order of the plurality of second analysis models according to the second directed acyclic graph; Acquire a plurality of second analysis tasks generated by the plurality of second analysis models, each second analysis model generating one second analysis task; Execute the second analysis task generated by each second analysis model according to the second topology sequence to obtain a second analysis result corresponding to each second analysis task; According to the second analysis result corresponding to each second analysis task, the analysis object where the fault event occurs is determined.

6. The method according to claim 1, characterized in that Before obtaining the first directed acyclic graph consisting of a plurality of first analysis models, the method further includes: The indicator collector uses different indicator collection methods to collect indicator data from different indicator data sources; Report the indicator data collected by the indicator collector through the indicator receiver and write it into the distributed message queue; The index data in the message queue is stored in a database.

7. The method according to claim 1, characterized in that Before obtaining the first directed acyclic graph consisting of a plurality of first analysis models, the method further includes: Use the log collector to collect and analyze the newly added log text content of the object in real time; Parsing the log text content according to the log type of the log text content through a log processor, extracting structured fields, and writing the structured fields into a distributed message queue; The structured fields in the message queue are stored in a database.

8. A big data cluster fault analysis device, characterized in that: The device comprises: A first acquisition module, used to acquire a first directed acyclic graph composed of multiple first analysis models, each node in the first directed acyclic graph corresponds to a first analysis model, and each first analysis model is used to analyze indicator data and / or log data of an analysis object in the cluster; A first determining module, configured to determine a first topological order of the plurality of first analysis models according to the first directed acyclic graph; A second acquisition module is used to acquire a plurality of first analysis tasks generated by the plurality of first analysis models, each first analysis model generates a first analysis task; an execution module, configured to execute the first analysis task generated by each first analysis model according to the first topological order, and obtain a first analysis result corresponding to each first analysis task; The second determination module is used to determine the fault analysis result of the analysis object corresponding to each first analysis task according to the first analysis result corresponding to each first analysis task.

9. An electronic device, characterized in that: The electronic device comprises: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the big data cluster fault analysis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images