Fault diagnosis method and equipment for dispatching system

By receiving and aggregating the metric data of the scheduling system and generating diagnostic reports, the problem of difficulty in job scheduling fault diagnosis in public cloud platforms is solved, and the fault diagnosis efficiency and user experience are improved.

CN114116428BActive Publication Date: 2024-11-26CHINA CONSTRUCTION BANK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202111459357.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-01
Publication Date
2024-11-26
Estimated Expiration
2041-12-01

AI Technical Summary

Technical Problem

There is difficulty in troubleshooting during job scheduling in public cloud platforms. The existing technology lacks scheduling indicator recording and fault analysis methods for a single job, resulting in high threshold for users and difficulty in positioning fault points in the scheduling process.

Method used

It provides a fault diagnosis method for scheduling system, by receiving and aggregating indicator data, obtaining target indicator data according to the diagnosis request type for processing, generating diagnostic reports, reducing the threshold for users to troubleshoot themselves and improving fault diagnosis efficiency.

Benefits of technology

It realizes effective diagnosis of job flow, job and scheduling service faults, simplifies user troubleshooting process, lowers the threshold for use, and improves the efficiency of fault diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114116428B_ABST
    Figure CN114116428B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a method and device for fault diagnosis of a scheduling system, the method comprising: receiving indicator data sent by the scheduling system, and uploading the aggregated indicator data to a first database after aggregating the received indicator data; upon receiving a diagnosis request sent by a user, obtaining a target diagnosis rule from a second database according to the diagnosis type corresponding to the diagnosis request; obtaining target indicator data required for diagnosis from the first database according to the diagnosis type, and processing the target indicator data according to the target diagnosis rule to obtain a diagnosis report; and sending the file address of the diagnosis report to the scheduling system. The present application can better display the fault points of job flows / jobs and scheduling services to users, making it easier for users to troubleshoot by themselves or contact operation and maintenance personnel in a timely manner for processing, lowering the usage threshold for platform users, and improving fault diagnosis efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of public cloud technology, and in particular to a fault diagnosis method and device for a scheduling system. Background Art

[0002] In terms of service objects and scope, cloud computing platforms can be divided into public clouds, private clouds and hybrid clouds. Among them, public cloud platforms generally respond to, allocate and resend received tasks through job scheduling systems.

[0003] Faced with the massive amount of tasks in the public cloud platform, some failures are inevitable during the job scheduling process, so it is necessary to analyze and diagnose the failures generated during the scheduling process at any time. When analyzing and diagnosing failures during the scheduling process, existing cloud platform users generally encounter the following problems: 1) The existing scheduling process has many steps, and the job flow and job state transfer relationship are complex. Any conversion process involves many key steps. Any error in any link will cause the job to fail to schedule and run normally, but the fault location method can only rely on checking the error points in the background log; 2) The existing operation indicators are all statistical indicators corresponding to the entire scheduling system. There is no indicator record for a single job, so it is difficult to aggregate the scheduling process records of a single job and find the failure points in the scheduling process; 3) The existing scheduling framework lacks the analysis method for scheduling service failures, resulting in a relatively high threshold for use.

[0004] Therefore, how to improve the fault diagnosis capability of job flows, jobs or scheduling services in the scheduling system is a technical problem that urgently needs to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method and device for fault diagnosis of a scheduling system, which can effectively improve the fault diagnosis capability of a job flow, job or scheduling service in the scheduling system.

[0006] In a first aspect, an embodiment of the present application provides a method for diagnosing a fault in a scheduling system, the method comprising:

[0007] Receiving the indicator data sent by the scheduling system, aggregating the received indicator data, and uploading the aggregated indicator data to the first database;

[0008] When receiving a diagnosis request sent by a user, acquiring a target diagnosis rule from a second database according to a diagnosis type corresponding to the diagnosis request, wherein the diagnosis type corresponding to the diagnosis request includes any one of a workflow diagnosis, a job diagnosis, and a scheduling service diagnosis;

[0009] Acquire target indicator data required for diagnosis from the first database according to the diagnosis type, and process the target indicator data according to the target diagnosis rule to obtain a diagnosis report;

[0010] The file address of the diagnostic report is sent to the scheduling system.

[0011] In a second aspect, an embodiment of the present application provides a fault diagnosis device for a scheduling system, comprising:

[0012] An indicator aggregation module, used for receiving indicator data sent by the scheduling system, and after aggregating the received indicator data, uploading the aggregated indicator data to the first database;

[0013] a diagnosis analysis module, configured to, upon receiving a diagnosis request sent by a user, obtain a target diagnosis rule from a second database according to a diagnosis type corresponding to the diagnosis request, wherein the diagnosis type corresponding to the diagnosis request includes any one of a workflow diagnosis, a job diagnosis, and a scheduling service diagnosis;

[0014] The diagnostic analysis module is further used to obtain target indicator data required for diagnosis from the first database according to the diagnostic type, and process the target indicator data according to the target diagnostic rule to obtain a diagnostic report;

[0015] The diagnostic analysis module is also used to send the file address of the diagnostic report to the scheduling system.

[0016] In a third aspect, the present application provides an electronic device, including: at least one processor and a memory;

[0017] The memory stores computer-executable instructions;

[0018] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the fault diagnosis method for the scheduling system provided in the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the fault diagnosis method of the scheduling system provided in the first aspect is implemented.

[0020] The present application provides a method for fault diagnosis of a scheduling system, which forms a record of job flow, job scheduling process and diagnosis of scheduling service by aggregating indicator data, and further analyzes and diagnoses the aggregated indicator data, so as to better display job flow / job and scheduling service fault points to users, facilitate users to troubleshoot by themselves or contact operation and maintenance personnel in time for processing, reduce the usage threshold of platform users, and improve fault diagnosis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A schematic diagram of a flow chart of a method for diagnosing a fault in a scheduling system provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the architecture of a fault diagnosis system for a scheduling system provided in an embodiment of the present application;

[0024] Figure 3 A work flow state flow diagram provided in an embodiment of the present application;

[0025] Figure 4 A working state flow diagram provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a work flow or work diagnosis and analysis process in an embodiment of the present application;

[0027] Figure 6 This is a schematic diagram of the service indicator diagnosis and analysis process in the embodiment of this application;

[0028] Figure 7 A schematic diagram of a program module of a fault diagnosis device for a scheduling system provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In addition, although the disclosure in the present application is introduced according to one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately.

[0031] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0032] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0033] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0034] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0035] The present application relates to a job scheduling system in a public cloud platform, and provides a method for index aggregation and fault analysis and diagnosis of scheduling business and scheduling services for massive job scheduling in the public cloud platform, thereby realizing intelligent diagnosis of the job scheduling process.

[0036] At present, cloud platform users generally encounter the following problems when analyzing and diagnosing faults in the scheduling process:

[0037] (1) The existing scheduling framework has many steps in the scheduling process. The job flow and job state transition relationship are complex. Any conversion process involves many key steps. Any error in any link will cause the job to fail to be scheduled and run normally. However, the fault location method can only check the error point in the background log.

[0038] (2) The operating indicators of the existing scheduling framework are all statistical indicators corresponding to the entire scheduling system. There are no indicator records for individual jobs, making it difficult to aggregate the records of the scheduling process of a single job and find the fault points in the scheduling process.

[0039] (3) The existing scheduling framework lacks the means to analyze and diagnose scheduling service failures, resulting in a high threshold for ordinary users to use the framework.

[0040] (4) Currently, heterogeneous and diverse computing platforms generate various types of jobs. The existing framework generally submits jobs of a specific platform to the corresponding computing platform, lacks job status monitoring, and the job status monitoring indicators of the existing framework are not exposed to the outside world. As a result, users need to associate the logs of a specific computing platform when troubleshooting, which makes operation and maintenance troubleshooting difficult.

[0041] Exemplarily, in some embodiments, the Apache Airflow scheduling system usually uses the StatsD framework to implement indicator aggregation and diagnostic analysis. In this technical solution, the Apache Airflow scheduling system is mainly divided into three components, Airflow WebServer (page server), Airflow Scheduler (backend scheduling engine) and Airflow Woker (job execution client). The codes of these three components are embedded with the creation of StatsD clients and the sending of response indicators to the StatsD server. The StatsD server is a daemon on the server that listens to statistical indicators in UDP and TCP requests and sends aggregation results to the backend database. After receiving the indicators, the StatsD server will parse the data and aggregate it, and then push it to the backend database at regular intervals, such as influxDB or ElasticSearch. The external monitoring and alarm system can request the backend database to obtain indicators to complete the monitoring and alarm function. The diagnostic analysis of this technical solution is relatively simple, and the overall scheduling situation can be analyzed by analyzing the indicator data, such as users can use tools such as Kibana or Grafana for analysis; in addition, users can draw diagnostic conclusions on service anomalies and job anomalies by analyzing logs.

[0042] In the above technical solution, the indicators supported by the Airflow scheduling system mainly include three types: counting type (Counters), scalar type (Gauges) and timer type (Timers). These three types of indicators basically reflect the situation at the global level of scheduling, and only the timer indicator can have a response record for a specific job. Among them, the counting type means that the indicator aggregation is in the form of counting sum. The Airflow scheduling system supports 26 counting indicators including the total number of successful / failed / queued jobs, the total number of running job flows, and other counting indicators; the scalar type means that there is no need to aggregate the indicator. If the indicator is not updated next time, the original value is maintained. The Airflow scheduling system supports 22 scalar indicators including the total number of running / queued jobs on the Worker service node, the total number of tasks that timed out in parsing dag files, etc. The timer type means that the indicator aggregation is in a way with a time window, which can represent the average value / total number / percentile of the specified direct characteristic data, etc. Airflow supports 9 timer indicators including specific task execution time, specific task scheduling delay, and the time taken for a specific task to execute to a successful state.

[0043] In addition, if the StatsD service needs to be horizontally expanded, the StatsD Proxy service needs to be deployed, which can ensure that the same indicator is sent to a fixed node.

[0044] The defects of the above technical solution are: the StatsD server needs to parse the data and aggregate indicators first. When the amount of data reported is too large, the scheduling framework needs to modify the sampling rate when sending, otherwise the server-side data processing will form a performance bottleneck; the user needs to determine the sampling rate based on experience, which has great limitations; when the StatsD server performs horizontal service expansion, it cannot solve the scenario where the traffic of specific indicators is too large and load balancing is required; the framework itself does not provide scheduling indicators corresponding to a single job flow or job, making it difficult to form abnormal diagnosis and analysis of job flows and jobs, and users need to check the logs themselves; the framework itself does not provide diagnostic analysis functions, and users need to analyze indicator data, even service logs and job logs, by themselves.

[0045] In other embodiments, the scheduling system can use the Apache Azkaban framework to implement task indicator collection and diagnostic analysis. In this technical solution, the Apache Azkaban scheduling system is mainly composed of three key components: WebServer, ExecutorServer and MySQL relational database. Among them, Azkaban ExecutorServer is mainly responsible for the submission and execution of specific job flows, and the scheduling capability can be expanded by starting multiple execution servers. ExecutorServer coordinates the execution of tasks through the MySQL database. In the process of submitting and executing job flows, ExecutorServer embeds the code for indicator aggregation in the code, writes the indicator into the MySQL database, and reports it to WebServer regularly. The Azkaban scheduling system will expose the access link of the corresponding indicator to external applications through WebServer, and external applications can obtain indicator data through http requests. However, the Azkaban framework currently only provides 5 indicator data, including the total number of successful / failed / queued job flows and the total number of running / failed jobs, and by modifying the http request parameters, aggregation by time window and by number of instances can be achieved.

[0046] In the above technical solution, the core component of the framework will provide both service logs and job logs to users, and users can analyze the log contents to diagnose anomalies.

[0047] The defects of the above technical solution are: the framework itself does not provide scheduling indicators corresponding to a single job flow or job, making it difficult to form abnormal diagnosis and analysis of job flows and jobs, and users need to check the logs themselves; the framework itself does not provide diagnostic analysis functions, and users need to analyze indicator data, even service logs and job logs, by themselves; the indicator data provided is relatively small, and if the external system requests indicator data frequently, it is easy to affect the processing performance of the core scheduling component.

[0048] In the face of the above technical problems, the present application provides a fault diagnosis method for a scheduling system, which forms a record of the job flow, job scheduling process and a diagnosis of the scheduling service by aggregating indicator data, and further analyzes and diagnoses the aggregated indicator data, which can better display the job flow / job and scheduling service fault points to users, making it easier for users to troubleshoot by themselves or contact operation and maintenance personnel in a timely manner for processing, thereby lowering the usage threshold for platform users and improving fault diagnosis efficiency.

[0049] In some embodiments, the existing scheduling system generally only provides aggregate indicators that express the overall situation of scheduling, and there are only a few or even no corresponding indicators to express a single job flow or job. Currently, there is little system support for the scheduling and operation indicators of jobs on various heterogeneous computing platforms. This situation has led to the fact that for existing scheduling systems, the means of fault location can basically only be checked in the service log or the error point in the job log. Based on the common aggregate indicator data, this application abstracts and expands the business indicators of each stage of scheduling corresponding to a single job flow or job, and forms the cleaning, pruning and processing of the indicator data by defining and constructing a scheduling business rule tree, and finally forms a diagnostic report for the corresponding job flow or job.

[0050] In some embodiments, in the face of existing scheduling systems, users are required to actively analyze service logs, or use tools such as Kibana or Grafana to visualize and analyze aggregated indicator data to analyze the diagnostic report of the scheduling service, which has a high threshold for users to use. This application defines and constructs service indicator rules, which is equivalent to a display template for aggregated indicator data. According to the template, the existing multiple indicator data are cleaned and processed, and finally an overall diagnostic report of the corresponding scheduling service is formed.

[0051] In some embodiments, facing the limited horizontal expansion capability of the indicator aggregation service in the prior art solution, the present application combines the indicator aggregation and diagnostic analysis functions into an indicator diagnosis module, which supports traffic load balancing and flexible expansion and contraction in the microservice architecture. The indicator diagnosis module is decoupled from the scheduling system through http, which will not directly affect the performance of the scheduling system, and the scheduling system can flexibly choose whether to enable this function through the configuration file.

[0052] The following is a detailed description using a detailed embodiment.

[0053] Reference Figure 1 , Figure 1 A schematic diagram of a flow chart of a method for diagnosing a fault in a dispatching system provided in an embodiment of the present application. Figure 1 As shown, the fault diagnosis method of the scheduling system includes:

[0054] S101, receiving indicator data sent by a scheduling system, performing aggregation processing on the received indicator data, and uploading the aggregated indicator data to a first database.

[0055] In the embodiment of the present application, an indicator diagnosis module can be pre-constructed, and the indicator diagnosis module is divided into two sub-modules, namely an indicator aggregation module and a diagnosis analysis module.

[0056] In a feasible implementation, each core component of the scheduling system can report indicator data to the indicator aggregation submodule in Http mode. After receiving the indicator data sent by the scheduling system, the indicator aggregation submodule parses the indicator data in the message, determines the specific indicator type, and performs aggregation calculations as needed.

[0057] Among them, the indicator aggregation module will have a timing thread to regularly push the indicator data to the first database on the backend.

[0058] Optionally, the first database may be a time series database (influxDB) or a non-relational database (ElasticSearch).

[0059] In some embodiments, scheduling process business indicator data, such as scheduling business indicators of job flows and jobs, will be stored in chronological order to facilitate filtering indicator data according to the start and end time of the job flow or job; for scheduling service indicator data, such as the total number of jobs running / successful / failed / queued, service heartbeat time, time required for job flow execution and other data, they are stored in specific data areas with corresponding indicator meanings to facilitate external service reading.

[0060] S102: When receiving a diagnosis request sent by a user, obtain a target diagnosis rule from a second database according to a diagnosis type corresponding to the diagnosis request.

[0061] In an embodiment of the present application, when a user initiates a diagnostic request to the scheduling system, the scheduling system will perform user authentication. If the user authentication is successful, the diagnostic request will be forwarded to the diagnostic analysis module. Otherwise, the diagnostic request will not be forwarded to the diagnostic analysis module.

[0062] After receiving the diagnosis request, the diagnosis analysis module parses the diagnosis type from the message, then obtains the diagnosis rule from the second database, and assembles a diagnosis rule tree or a diagnosis rule list.

[0063] Optionally, the second database is a relational database.

[0064] The above diagnosis types generally include: job flow diagnosis, job diagnosis and scheduling service diagnosis.

[0065] S103, acquiring target indicator data required for diagnosis from the first database according to the diagnosis type, and processing the target indicator data according to the target diagnosis rule to obtain a diagnosis report.

[0066] In the embodiment of the present application, target indicator data required for diagnosis is obtained from the first database according to the diagnosis type.

[0067] In some embodiments, when the diagnosis type corresponding to the above-mentioned diagnosis request is scheduling service diagnosis, the target indicator data currently uploaded to the first database is obtained; when the diagnosis type corresponding to the above-mentioned diagnosis request is job flow diagnosis or job diagnosis, the target indicator data uploaded within the specified time range in the first database is obtained according to the job flow running start time in the diagnosis request.

[0068] The specified time range is from the start time of the workflow in the diagnosis request to the time when the diagnosis analysis module receives the diagnosis request.

[0069] In some embodiments, when the above-mentioned diagnosis type is workflow diagnosis or job diagnosis, the acquired diagnosis rules are assembled into a diagnosis rule tree, and the acquired indicator data is cleaned, trimmed and processed according to the diagnosis rule tree to generate a diagnosis report; when the above-mentioned diagnosis type is scheduling service diagnosis, the acquired diagnosis rules are assembled into a diagnosis rule list, and the acquired indicator data is cleaned, trimmed and processed according to the diagnosis rule list to generate a diagnosis report.

[0070] S104: Send the file address of the diagnosis report to the scheduling system.

[0071] In the embodiment of the present application, after the diagnosis report is generated, the diagnosis report file address is returned to the scheduling system. After the scheduling system asynchronously obtains the diagnosis report file address, it reads the corresponding content and displays it on the front-end page for the user to view.

[0072] In addition, users can access the web interface of the diagnostic analysis submodule to upload data in JSON format according to the iterative evolution of the scheduling system, and can modify and add or subtract existing diagnostic rules.

[0073] A fault diagnosis method for a scheduling system provided in an embodiment of the present application forms a record of job flows, job scheduling processes, and a diagnosis of scheduling services by aggregating indicator data, and further analyzing and diagnosing the aggregated indicator data, thereby better displaying job flows / jobs and scheduling service fault points to users, making it easier for users to troubleshoot by themselves or contact operation and maintenance personnel in a timely manner for processing, thereby lowering the usage threshold for platform users and improving fault diagnosis efficiency.

[0074] Reference Figure 2 , Figure 2 A schematic diagram of the architecture of a fault diagnosis system for a scheduling system provided in an embodiment of the present application.

[0075] In some embodiments, the indicator aggregation module is responsible for receiving the indicator data sent by the scheduling system, performing aggregation operations as needed, and sending the index data to the first database at regular intervals.

[0076] Among them, the above indicator data comes from the logical code of the scheduling system, and it is necessary to embed the logic of indicator sending in the code logic.

[0077] In some embodiments, the message for sending indicator data includes the following information: metricName (indicator name), metricTime (indicator sending time), metricType (indicator type), operator (indicator operand), operatorElement (indicator operation object), operatorResult (indicator operation result), instId (instance id), instStatus (instance status), zxid (globally unique id), metricContext (indicator content), errCauseReason (operation step error reason), errMsg (error specific information), module (scheduling module name), hostIP (module network IP), etc.

[0078] Since the indicators are divided into scheduling business indicators and scheduling service indicators, the difference is that scheduling business indicators tend to record process results, so it is necessary to add the instance status and instance ID corresponding to the operation, while service indicators tend to count and record the latest information, which is processed by the indicator aggregation module, so there is no need to fill in the operation error and corresponding instance information.

[0079] In some embodiments, the scheduling service indicators are further divided into job flow scheduling service indicators and job scheduling service indicators, while the scheduling service indicators are generally divided into counting type, scalar type and timing type according to the aggregation mode. Therefore, although there are only three different indicators in the business, metricType needs to be mapped to 5 types, corresponding to 5 values.

[0080] When the indicator aggregation submodule receives a message, it needs to first verify the indicator type, and then verify whether the message information is correct and complete based on the indicator type, and then perform different logical processing on the scheduling business indicators and scheduling service indicators respectively.

[0081] It should be noted that, firstly, since the core components of the scheduling system may be deployed in a distributed manner, a globally unique ID needs to be sent for distinction, and the module name and module network IP are also used for distinction; secondly, errors may be reported during the scheduling operation, so additional error causes and specific information of the operation steps are required; finally, regarding the indicator content, for business indicators it will be process record information, and for service indicators it will be the value of the corresponding indicator.

[0082] Among them, the scheduling business indicators are the tracking record indicator data of the whole process of job flows and job instances from creation to destruction in the scheduling system. In the scheduling system, there are many states of job flows and jobs, and there are many situations when the states flow. For the job flow state flow diagram, there are 7 job flow states, and there are 8 situations of state flow. For the job state flow diagram, there are 11 job states, and there are 14 situations of job state flow. For different scheduling systems, the job flows and job state flow diagrams are relatively common, but the specific intermediate steps involved in the state flow are very different.

[0083] In order to better understand the embodiments of the present application, refer to Figure 3 and Figure 4 , Figure 3 A work flow state flow diagram provided in an embodiment of the present application; Figure 4 This is a job status flow diagram provided in an embodiment of the present application.

[0084] In the embodiment of the present application, the operation of the abstract business indicator includes operands, operation objects and operation results. In order to cover the technical capabilities of intermediate operations of different systems as much as possible, such as files, message queues, relational databases, Redis, Kubernetes, Yarn, memory queues and Zookeeper, a total of 15 operation objects and corresponding 19 operands and corresponding operation results are abstracted.

[0085] When the business-critical intermediate steps in the scheduling system can be sorted out, and the scheduling system reports the indicators, it is only necessary to use the algorithm to match the order of the indicators and the intermediate steps to obtain a report on the entire scheduling process of a job flow or job instance, and to simply process or analyze it to obtain abnormal points. The sorted deterministic key intermediate steps can be represented as diagnostic rules by the abstract operation here, and by constructing a diagnostic rule tree, it is possible to represent various possible results after the state flow operation in the entire scheduling process.

[0086] In order to better understand the embodiments of the present application, refer to Table 1, which is a schematic diagram of the operands, operation objects and operation results corresponding to the indicators.

[0087] Table 1: Schematic diagram of the number of operations, operation objects and operation results corresponding to the indicators

[0088]

[0089] Different modules in the scheduling system execute different intermediate key steps in the job flow and job status flow diagram. After the execution is completed, the corresponding execution results or exceptions need to add the logic of sending business indicators. For example, after receiving the job information, the execution module responsible for the job operation in the scheduling system needs to send an http request to the indicator aggregation submodule after parsing the job parameters and successfully submitting it to Kubernetes to create a job running instance. The request contains information such as the operand is submit, the operation object is Kubernetes, the operation result is successful, and it carries the instance id and the instance status is running, and the corresponding indicator content is successfully submitting the job to Kubernetes and creating a job instance. The subsequent operation module also needs to continuously request the Kubernetes Api Server service to obtain the running status of the job instance. Each time the monitoring data is obtained, it needs to send another http request to the indicator aggregation submodule, and the redundant indicator content will be cleaned during the diagnosis and analysis.

[0090] In some embodiments, after the indicator aggregation module receives the http request, it first turns to the processing flow of other indicator types according to the indicator type. If it is not a business indicator data, it verifies the integrity of its information, parses the indicator data, calls the business indicator processing service and converts it into the Append command of the service processing queue, and adds it to the queue. If the queue is added successfully, the request is returned successfully, otherwise it fails. The business indicator processing service contains a Sender instance, which will select whether the Sender instance sends data to ElasticSearch or influxDB according to the configuration file. Secondly, the business indicator processing service will have a thread that continuously takes out the processing command from the processing queue. When the processing command is Append, the indicator data will be added to the memory queue of the Sender instance, and it will be placed in the memory queue of the job flow or job according to the business indicator type. Furthermore, the business indicator processing service will also start a timing thread. The timing interval is the refresh interval of the business indicator in the configuration file to send the back-end database, and the interval puts the Flush command into the processing queue. Finally, when a thread takes out the Flush command from the processing queue, it will first check whether the data area exists. If it does not exist, it will be created first. For the ES database, it will create an index. If the data area exists, the data in the Sender instance memory queue will be asynchronously written to the corresponding data area of ​​the backend database by calling the database corresponding API in batches. For the naming of the data area, the user can set a prefix in the configuration file. The business indicator will use the date as a suffix and create a data area with this name. Finally, when the business processing indicator service is to be closed, it is necessary to first add a flush command and then a close command to the processing queue. If the thread of the business indicator processing service processes the close command, the polling operation of the queue processing thread of the indicator processing service will be stopped, and the timing thread will be waited for to exit together, and the connection of the backend processing library will be closed at the same time.

[0091] In some embodiments, the scheduling service indicator is a performance-related indicator of the scheduling service in the scheduling system, which is mainly divided into three types: scalar type, counting type and timing type. These three types are distinguished according to the indicator type in the message. When the scheduling system sends the scheduling service indicator, the instance id and instance status are not required in the message.

[0092] It should be noted that, since a set of message structures are multiplexed, the indicator content field represents the recorded operation process information in the scheduling business indicator, while it represents the value of the corresponding indicator in the service indicator.

[0093] In some embodiments, depending on the scheduling system, the user can customize different scheduling service indicators. Only the message structure of the service indicator is needed, and the indicator aggregation method is confirmed, and sent to the indicator aggregation submodule in the form of http, so that the indicator can be recorded in the data area configured by the user in the configuration file. For example, the heartbeat time of the execution module of the scheduling system, the user needs to display the latest time in the diagnostic report, so it can be configured as a scalar type, and the code is embedded in the execution module in the scheduling system to report the indicator, and the indicator name is specified in the message as the heartbeat time of the execution module, and the heartbeat time is reported in the indicator content, and the operation object is Time. Furthermore, if the user needs to display the total number of successful execution job instances of the scheduling system in the diagnostic report, it can be configured as a counting type, and the counting indicator is reported every time the execution module completes a job, and the indicator name is specified in the message as the total number of successful execution job instances of the execution module, and the indicator content is 1, and the operation object is NUM. When the service indicator is checked, whether the operation object is NUM and TIME, if it is other objects, it will directly return failure and discard the data.

[0094] In some embodiments, after the indicator aggregation module receives the http request, it first turns to the processing flow of other indicator types according to the indicator type. If it is a service indicator, it verifies the integrity of its information, parses the indicator data, calls the service indicator processing service and converts it into the Append command of the processing queue. If the service indicator type is a counting type or a timing type, it needs to add a Compute command to the processing queue. If all queues are added successfully, the request is returned to be successful, otherwise it fails. The service indicator processing service contains a Sender instance, which will choose whether the Sender instance sends data to ElasticSearch or influxDB according to the configuration file.

[0095] Secondly, the service indicator processing service will have a thread that continuously takes out processing commands from the processing queue. When the processing command is Append, the indicator data will be added to the memory structure of the Sender instance. This is divided into three cases. In the Sender instance, a Map is maintained for counting indicators. The key value is the indicator name and the value is a List. For the Append command, you only need to add the indicator value to the List; two Maps are also maintained for timing indicators. The key values ​​of these two Maps are the indicator names. The value of one of them is a fixed-length Queue. For the Append command, you only need to add the indicator value to the Queue, and the other is a Pair, which is stored after the calculation is completed. The indicator corresponds to the total duration and the number of accumulated indicators calculated in memory; a Map is also maintained for scalar indicators. The key value is the indicator name and the value is the indicator value. The Append instruction will directly modify the value to the current value. When the processing command is Compute, only for the timing type, it is relatively special that there are abnormal extreme values ​​in the time range. For example, when a service is abnormal, the time consumed from job creation to completion will be much longer than normal. Therefore, it is necessary to sort the data in the Queue, remove the 10% before and after, and then accumulate them. The accumulated results will be stored in the memory.

[0096] Furthermore, the business indicator processing service will start a timing thread. The timing interval is the refresh interval for sending the business indicators in the configuration file to the backend database. The SingleFlush command and ReadFlush command are placed in the processing queue at intervals. Then, when a thread takes out the SingleFlush command and ReadFlush command from the processing queue, the SingleFlush command will only process scalar indicators, and ReadFlush will only process count and timing indicators. Both will first check whether the data area exists. If it does not exist, it will be created first. For the corresponding ES database, it will create an index. If the data area exists, the SingleFlush command asynchronously writes the data in the Sender instance's scalar memory Map to the corresponding data area of ​​the backend database one by one by calling the corresponding database API. The ReadFlush command will first process the counting indicators, that is, loop the Map corresponding to the counting type maintained in the memory of the Sender instance, and use the optimistic locking mechanism for each indicator to read the corresponding cumulative value from the data area, add it to the elements in the List to get the new value and update it to the data area, clear the List at the same time, and then process the timing indicators. Also use the optimistic locking mechanism to read the value of the corresponding timing indicator from the data area, and get the total duration and count value, which are accumulated with the total duration and count value in the memory Map.

[0097] When the business processing indicator service is to be closed, two flush commands (i.e. SingleFlush and ReadFlush commands) and a close command need to be added to the processing queue first. If the thread of the business indicator processing service processes the close command, the polling operation of the queue processing thread of the indicator processing service will be stopped, and the timing thread will be waited for to exit together, and the connection of the backend processing library will be closed at the same time.

[0098] It should be noted that, in terms of service indicator aggregation, although the aggregation mode of the present invention is basically the same as that of StatsD, the way of aggregation calculation is essentially different from that of StatsD. The reason is that when StatsD is deployed in a cluster, it relies on the StatsDproxy service. In order to be compatible with the logic of a single service, this service sends an indicator to a fixed node, simply to avoid concurrency problems when writing to multiple nodes. However, when the request traffic for a specific indicator surges, even if N service nodes are expanded, there is no way to achieve load balancing of the traffic. The present invention, regarding counting and timing service indicators, introduces optimistic locking when writing to multiple nodes concurrently, which can achieve a balance between performance and scalability to a large extent. In the case of a surge in traffic, it can also directly support load balancing of traffic in the case of horizontal expansion.

[0099] In some embodiments, due to the complexity of the job flow and job status flow in the scheduling system, before performing diagnostic analysis, the operations of key intermediate steps in the status flow are abstracted in the indicator reporting stage.

[0100] In the abstract operation, this application defines 15 kinds of operation objects and 19 kinds of operands and corresponding operation results. When defining the diagnosis rules, the diagnosis rules can simply be written as the grammar of the abstracted operation objects, operands and corresponding operation results, and thus express the intermediate steps in each state transition process, or express a constraint relationship.

[0101] For example, it is known that the transfer of the job status from waiting for resources to canceling requires a termination operation in manual intervention, and the key step of the termination operation is to change the job status from waiting for resources to termination in the relational database, so the diagnostic rule can be written as update DB_JOB_STATUS SUCCESS; when the job status is diagnosed as waiting for resources, the same update DB_JOB_STATUS object can be compared according to the operation information of the business indicator data to see whether the operation result is SUCCESS. If it is FAILED, then the fault point is indicated in the diagnostic report. Furthermore, it is known that the heartbeat reporting time of each module in the scheduling system cannot exceed 10 minutes without being updated, so the diagnostic rule can be written as less_than TIME 10, indicating that it is 10 minutes less than the current time; when diagnosing the heartbeat reporting time of all services, the heartbeat time in the indicator content can be obtained according to the indicator content of the service indicator data. At this time, according to the operation calculation of the diagnostic rule, the heartbeat time is compared with the current time to see whether it is greater than the constraint value of 10 minutes, and it can be diagnosed whether the requirements are met, so as to determine whether there is an abnormality.

[0102] Among them, the construction of the diagnostic rule tree is to restore the flow process of the entire scheduling system, and embed the abstracted diagnostic rules into the state flow diagram to obtain a diagnostic rule tree. The tree structure of this diagnostic rule tree and the node relationship of the tree structure can store it in a relational database. Therefore, the present application constructs a table diagnose_rules in a relational database, and includes the following fields: rule_id (rule id), rule_detail (rule content details), rule_suti_condition (rule applicability conditions), parent_rule_id (node ​​parent node rule id), level (tree level, 0 represents the root node), err_tips (abnormal information prompt), info_tips (normal information), rule_type (rule category, 1 represents the job flow diagnostic rule, 2 represents the job diagnostic rule, and 3 represents the service diagnostic rule). The rule content details are the written rules themselves, and the rule usage conditions are the matching expressions of the service indicator names, or the status of the job flow / job. One additional point that needs to be explained is that, considering that service indicator information essentially has no state flow relationship, the level of service diagnosis rules is all flat, and the default value is -1.

[0103] The processes of job flow and job diagnosis analysis are basically the same. The only difference is that the diagnosis rule tree stored in the database is different and the state flow is also different.

[0104] Reference Figure 5 , Figure 5It is a schematic diagram of the operation flow or operation diagnosis and analysis process in the embodiment of the present application.

[0105] In some embodiments, the above-mentioned operation flow or operation diagnosis and analysis process includes:

[0106] S501: Determine whether the diagnosis request contains an instance ID, a start time, and an end time, where the instance ID, the start time, and the end time are of a workflow instance or a job instance.

[0107] In some embodiments, the diagnosis type and the instance ID, start time, and end time are verified in the diagnosis request. The time range is given to obtain data from different data areas based on the range. For example, the business indicator data of the job instance may be stored in the index of multiple time dates in ElasticSearch, so the range needs to be determined.

[0108] S502: If the diagnosis request contains an instance ID, a start time, and an end time, traverse each node of the diagnosis rule tree, and traverse the target indicator data at each traversed node.

[0109] In some embodiments, if the diagnostic request contains an instance id, a start time, and an end time, the diagnostic rules are obtained from the second database, and a diagnostic rule tree is assembled according to the obtained diagnostic rules; if the diagnostic request does not contain an instance id, a start time, and an end time, the request fails and an error message is displayed.

[0110] In some embodiments, indicator data is obtained from the first database according to instance status, instance id, start time and end time and converted into a list. The List can be sorted according to zxid. Since the larger the zxid, the newer the indicator, deduplication is then performed based on the operation object, operation number and operation result.

[0111] S503: If the instance state of the indicator in the target indicator data satisfies the diagnosis rule of the currently traversed node, the currently traversed node is added to the diagnosis result list.

[0112] In some embodiments, all job flow / job instance status values ​​are traversed, where the traversal order is determined by the flow order.

[0113] According to the breadth-first search algorithm of the tree, traverse some nodes of the diagnosis rule tree. Here, you need to initialize a partQueue, which contains the root node of the instance state of the current cycle in the diagnosis rule tree. Then, using the breadth-first search algorithm, traverse all indicator data of the instance for each node. If the instance state of an indicator is consistent with the rule application conditions of the current node, and the operation of the indicator is consistent with the rule details of the current node, then add the node to the diagnosis result list.

[0114] If the indicator has errMsg or errCausedReason, the level needs to be converted to ERROR when the indicator is output. Otherwise, the level defaults to INFO, and the indicator content, that is, the scheduling operation record, is displayed.

[0115] S504: Generate the diagnosis report according to the diagnosis result list.

[0116] For each instance status, a breadth-first search is performed and finally a list of diagnostic results is obtained. The list of diagnostic results needs to be traversed. For the indicator data of job monitoring, only the first and last monitoring results need to be retained to beautify the display.

[0117] Write all indicator results into the file in order according to "indicator reporting time, information level, indicator content", then return the request success and attach the file address to the scheduling system.

[0118] After the above steps, the final job flow / job diagnosis report format is as follows:

[0119] 2021-0615 18:58:11.367INFO Pulling job dependency verification event message successfully;

[0120] 2021-0615 18:58:11.368INFO The job pre-dependencies are ready, and the job example status is updated to ready;

[0121] 2021-0615 18:58:11.369INFO The job is not pending, ready to be pushed to the job priority queue;

[0122] 2021-0615 18:58:11.369INFO The job instance was converted into a priority object, and the priority object was successfully placed in the Redis priority queue;

[0123] 2021-0615 18:58:12.007INFO Successfully obtained tenant lock;

[0124] …

[0125] In some embodiments, compared with the job flow / job diagnostic analysis, the diagnostic rules of the scheduling service diagnostic analysis do not have a tree-like hierarchical relationship. Figure 6 , Figure 6 Schematic diagram of the service indicator diagnosis and analysis process in the embodiment of the present application.

[0126] In some embodiments, the above-mentioned operation flow or operation diagnosis and analysis process includes:

[0127] S601: Determine whether there is a scheduling module name in the diagnosis request.

[0128] In some embodiments, when a diagnostic request is received, the diagnostic type and whether the diagnostic request contains a specific module name are first verified. If so, the service diagnostic analysis only analyzes the indicator data specific to the module, otherwise all service indicator data are analyzed.

[0129] S602: When the diagnosis request contains a scheduling module name, traverse each diagnosis rule in the diagnosis rule list, and traverse the target indicator data for each diagnosis rule traversed.

[0130] In some embodiments, the diagnosis rules may be obtained from the second database and converted into a diagnosis rule list. At the same time, all service diagnosis rules may be obtained from the database and converted into a list.

[0131] Traverse all service diagnosis rules, and for each rule, traverse all service indicator data. If the expression of the rule condition matches the current indicator name, and according to the diagnosis rule, it can be concluded that the service indicator data satisfies the rule constraint relationship, then the indicator is recorded in the diagnosis result list, and the info_tips of the rule is added to the indicator content. Otherwise, the indicator is recorded in the result List, and the output level is changed from the default INFO to ERROR, and the err_tips of the rule is added to the indicator content.

[0132] S603: If the currently traversed service indicator in the target indicator data meets the currently traversed diagnosis rule, the currently traversed service indicator is added to a preset diagnosis result list.

[0133] S604: Generate the diagnosis report according to the diagnosis result list.

[0134] In some embodiments, after the traversal is completed, all indicator data in the diagnosis result list are written into a file, and then the request is returned successfully with the file address attached.

[0135] After the above steps, when the diagnosis request is for the scheduling execution machine, the final diagnosis report about the scheduling execution machine is as follows:

[0136] 2021-0615 18:57:12.156INFO Execution module heartbeat report is normal executor_heartbeat_time2021-06-15 18:57:08;

[0137] 2021-0615 18:58:16.303INFO Execution module heartbeat report is normal executor_heartbeat_time2021-06-15 18:58:10;

[0138] 2021-0615 18:58:12.231INFO The average execution time of the execution module running instance is less than 30ms executor_job_run_time 2021-06-15 18:58:10;

[0139] …

[0140] In some embodiments, under the microservice architecture, the scheduling system and the indicator diagnosis module can be deployed in the container environment of Kubernetes, and the Nginx service is added to perform traffic load balancing and forwarding. Since the scheduling system and the indicator diagnosis module are decoupled through http, that is, both indicator reporting and diagnostic requests will pass through Nginx, for the scheduling system, the same service domain name is accessed, but it is not known which specific indicator diagnosis module is accessed. Each indicator diagnosis module is independent of each other and logically consistent, and accesses the same influxDB or ElasticSearch backend database. When the traffic is large, the traffic bandwidth will exceed the pre-set threshold, at which time the AutoScale mechanism pre-configured by Kubernetes will be triggered to automatically expand the indicator diagnosis module horizontally. When the traffic is small, the mechanism will horizontally shrink the module.

[0141] In this application, users only need to assemble data in json format according to the field format of the database table and confirm the content of the diagnostic rules by themselves. When the interface is requested for the first time, there is no data in the relational database, so the rule data in the message sent by the user will be directly parsed as initialization data. When the interface is requested subsequently, the rule data in the message sent by the user will be traversed. If it does not exist in the data table, it will be inserted into the data table. If it exists, the library table will be updated based on the message information sent by the user.

[0142] The indicator aggregation method for the scheduling system provided in this application has the following advantages:

[0143] In addition to the common aggregate indicators, the business indicators of a single job flow or each stage of job scheduling are abstracted and expanded, and the indicator data is cleaned, pruned and processed by defining and building a scheduling business rule tree, and finally a diagnostic report for the corresponding job flow or job is formed.

[0144] By defining and constructing a service indicator rule tree, it is equivalent to a display template for aggregated indicator data. According to the template, the existing various indicator data are cleaned and processed, and finally an overall diagnostic report for the corresponding scheduling service is formed.

[0145] It supports flexible expansion of service nodes based on traffic volume under the microservice architecture, and is decoupled from the scheduling system through HTTP to reduce the impact on scheduling performance.

[0146] The scheduling system is supported to flexibly turn on and off whether to report indicators and support diagnostic analysis by modifying configuration files.

[0147] Based on the contents described in the above embodiments, the present application also provides a fault diagnosis device for a scheduling system, referring to Figure 7 , Figure 7 This is a schematic diagram of a program module of a fault diagnosis device for a dispatching system provided in an embodiment of the present application. The fault diagnosis device for the dispatching system includes:

[0148] The indicator aggregation module 701 is used to receive the indicator data sent by the scheduling system, aggregate the received indicator data, and upload the aggregated indicator data to the first database.

[0149] The diagnosis analysis module 702 is used to obtain a target diagnosis rule from the second database according to a diagnosis type corresponding to the diagnosis request when receiving a diagnosis request sent by a user, wherein the diagnosis type corresponding to the diagnosis request includes any one of a workflow diagnosis, a job diagnosis, and a scheduling service diagnosis;

[0150] Acquire target indicator data required for diagnosis from the first database according to the diagnosis type, and process the target indicator data according to the target diagnosis rule to obtain a diagnosis report;

[0151] The file address of the diagnostic report is sent to the scheduling system.

[0152] It should be noted that the specific execution contents of the indicator aggregation module 701 and the diagnosis analysis module 702 in the embodiment of the present application can refer to the relevant contents in the above embodiments, which will not be repeated here.

[0153] Furthermore, based on the contents described in the above embodiments, an electronic device is also provided in an embodiment of the present application, which includes at least one processor and a memory; wherein the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory to implement the various steps of fault diagnosis of the scheduling system as described in the above embodiments, which will not be repeated in this embodiment.

[0154] In order to better understand the embodiments of the present application, refer to Figure 8 , Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0155] like Figure 8 As shown, the electronic device 80 of this embodiment includes: a processor 801 and a memory 802; wherein:

[0156] Memory 802, used to store computer-executable instructions;

[0157] The processor 801 is used to execute the computer-executable instructions stored in the memory to implement the various steps of fault diagnosis of the dispatching system described in the above embodiment, which will not be described in detail in this embodiment.

[0158] Optionally, the memory 802 may be independent or integrated with the processor 801 .

[0159] When the memory 802 is independently provided, the device further includes a bus 803 for connecting the memory 802 and the processor 801 .

[0160] Furthermore, based on the contents described in the above embodiments, a computer-readable storage medium is also provided in an embodiment of the present application, in which computer execution instructions are stored. When the processor executes the computer execution instructions, the various steps of fault diagnosis of the scheduling system as described in the above embodiments are implemented, and this embodiment will not be repeated here.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0162] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional module in each embodiment of the present application can be integrated into a processing unit, each module can exist physically separately, or two or more modules can be integrated into one unit. The above module integration unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0164] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application.

[0165] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0166] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0167] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0168] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0169] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0170] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A fault diagnosis method for a dispatching system, characterized in that: The method comprises: Receiving the indicator data sent by the scheduling system, aggregating the received indicator data, and uploading the aggregated indicator data to the first database; When receiving a diagnosis request sent by a user, acquiring a target diagnosis rule from a second database according to a diagnosis type corresponding to the diagnosis request, wherein the diagnosis type corresponding to the diagnosis request includes any one of a workflow diagnosis, a job diagnosis, and a scheduling service diagnosis; When the diagnosis type corresponding to the diagnosis request is the workflow diagnosis or the job diagnosis, according to the workflow running start time in the diagnosis request, obtaining the indicator data uploaded within the specified time range from the first database as the target indicator data required for diagnosis; Determine a state flow process of the scheduling system, and determine a state flow diagram of the scheduling system according to the state flow process, wherein the state flow diagram is a tree structure; Adding the target diagnostic rules to each node in the state flow diagram to generate a diagnostic rule tree; According to the diagnosis rule tree, the target indicator data is processed to obtain a diagnosis report; When the diagnosis type corresponding to the diagnosis request is the scheduling service diagnosis, obtaining the most recently uploaded indicator data from the first database as the target indicator data; Generate a diagnosis rule list according to the target diagnosis rule, and process the target indicator data according to the diagnosis rule list to obtain a diagnosis report; The file address of the diagnostic report is sent to the scheduling system.

2. The method according to claim 1, characterized in that The step of processing the target indicator data according to the diagnosis rule tree to obtain a diagnosis report includes: Determine whether the diagnosis request contains an instance ID, a start time, and an end time, wherein the instance ID, the start time, and the end time are of a job flow instance or a job instance; If the diagnosis request contains an instance ID, a start time, and an end time, traverse each node of the diagnosis rule tree, and traverse the target indicator data at each traversed node; If the instance state of the indicator in the target indicator data satisfies the diagnosis rule of the currently traversed node, the currently traversed node is added to the diagnosis result list; The diagnosis report is generated according to the diagnosis result list.

3. The method according to claim 1, characterized in that The step of processing the target indicator data according to the diagnostic rule list to obtain a diagnostic report includes: Determining whether a scheduling module name exists in the diagnostic request; When the diagnosis request contains a scheduling module name, traverse each diagnosis rule in the diagnosis rule list, and traverse the target indicator data for each diagnosis rule traversed; If the currently traversed service indicator in the target indicator data meets the currently traversed diagnosis rule, the currently traversed service indicator is added to the preset diagnosis result list; The diagnosis report is generated according to the diagnosis result list.

4. The method according to claim 1, characterized in that The receiving indicator data sent by the scheduling system includes: Receive a message sent by the scheduling system, wherein the message includes at least one of the following information: indicator name, indicator sending time, indicator type, indicator operand, indicator operation object, indicator operation result, instance ID, instance status, globally unique ID, indicator content, operation step error reason, error specific information, scheduling module name, module network IP; The indicator data is parsed according to the message.

5. The method according to claim 1, characterized in that The index data includes dispatching business index data and dispatching service index data; The scheduling business indicator data includes the tracking point record indicator data of the entire process from creation to destruction of the job flow and job instance in the scheduling system; The scheduling service indicator data includes performance indicator data of the scheduling service in the scheduling system. The types of the scheduling service indicator data in the scheduling system include scalar type, counting type and timing type.

6. The method according to claim 1, characterized in that Also includes: According to the received diagnosis rule message, a new diagnosis rule is added to the second database, or the diagnosis rule stored in the second database is modified.

7. A fault diagnosis device for a dispatching system, characterized in that: include: An indicator aggregation module, used for receiving indicator data sent by the scheduling system, and after aggregating the received indicator data, uploading the aggregated indicator data to the first database; a diagnosis analysis module, configured to, upon receiving a diagnosis request sent by a user, obtain a target diagnosis rule from a second database according to a diagnosis type corresponding to the diagnosis request, wherein the diagnosis type corresponding to the diagnosis request includes any one of a workflow diagnosis, a job diagnosis, and a scheduling service diagnosis; The diagnostic analysis module is further configured to, when the diagnostic type corresponding to the diagnostic request is the workflow diagnosis or the job diagnosis, obtain, from the first database, the index data uploaded within a specified time range as target index data required for diagnosis according to the workflow running start time in the diagnostic request; Determine a state flow process of the scheduling system, and determine a state flow diagram of the scheduling system according to the state flow process, wherein the state flow diagram is a tree structure; Adding the target diagnostic rules to each node in the state flow diagram to generate a diagnostic rule tree; According to the diagnosis rule tree, the target indicator data is processed to obtain a diagnosis report; When the diagnosis type corresponding to the diagnosis request is the scheduling service diagnosis, obtaining the most recently uploaded indicator data from the first database as the target indicator data; Generate a diagnosis rule list according to the target diagnosis rule, and process the target indicator data according to the diagnosis rule list to obtain a diagnosis report; The diagnostic analysis module is also used to send the file address of the diagnostic report to the scheduling system.

8. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the fault diagnosis method for the dispatching system according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the fault diagnosis method for the dispatching system according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Method and device for analyzing network equipment fault

    CN102137416A

  • Application monitoring method and device

    CN106528390A

  • Big data analysis model algorithm type selection method and device, electronic equipment and medium

    CN110990461A

  • A method and system for determining fault of heterogeneous system based on machine learning

    CN111209131A

  • Comprehensive analysis method and system for supervision data, medium and equipment

    CN113485989A