Abnormal link analysis method, device, equipment and storage medium

Abstract

The embodiment of the present application provides an abnormal link analysis method, device, equipment and storage medium, which receives a user's abnormal link analysis request, obtains the current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed included in the abnormal link analysis request, first determines whether the task to be processed is abnormal according to the current operation information, historical operation information and a pre-configured service level agreement, and when it is determined that the task to be processed is abnormal, recursively analyzes whether the task of the previous level of the task to be processed is abnormal based on the task hierarchy information, until the task hierarchy of the task without abnormality is determined, and the abnormal link analysis result is obtained. In this technical solution, the analysis of abnormal links can be performed based on the needs of the user, and the cause of the abnormal link can be located, which improves the accuracy of task monitoring and provides implementation conditions for improving the data quality output by the big data platform.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and in particular to an abnormal link analysis method, device, equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, various enterprises have begun to pay attention to the construction of big data platforms, using big data platforms to process and process various types of data to obtain the required data. In order to ensure that the data output by the big data platform meets the quality requirements, it is necessary to monitor the tasks running in the big data platform to ensure that the big data platform outputs timely and accurate data.

[0003] In the related technology, the task monitoring in the big data platform is mainly to set some alarm attributes for each task by grading, labeling, setting service level agreement (SLA) and other operations. Then, in actual application, when analyzing a task, the alarm attribute of the task and the historical operation status of the task can be combined to analyze the abnormality of the task, and an alarm notification will be issued when the task is determined to be abnormal. Specifically, a running time threshold is set for each task, and then the monitored tasks are scanned periodically to determine whether the running time of the monitored task exceeds the threshold time. If so, an alarm is triggered.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: when a monitored task triggers an alarm because its running time does not meet the time threshold, it is very likely that it is not caused by the abnormality of the task itself, but by the running abnormality of one or more parent tasks of the current task. The existing task monitoring method cannot directly locate the real cause of the task abnormality alarm, and there is a problem of low task monitoring accuracy. Summary of the invention

[0005] The embodiments of the present application provide an abnormal link analysis method, apparatus, device and storage medium to solve the problem of low task monitoring accuracy in existing task monitoring methods.

[0006] According to a first aspect of the present application, an embodiment of the present application provides an abnormal link analysis method, including:

[0007] Receive a user's abnormal link analysis request, the abnormal link analysis request including: an identifier of a task to be processed;

[0008] According to the identifier of the task to be processed, obtaining the current running information and the historical running information of the task to be processed;

[0009] Determining whether the task to be processed is abnormal according to the current operation information, the historical operation information and a pre-configured service level agreement;

[0010] When it is determined that the task to be processed is abnormal, based on the task level information, recursively analyze whether the previous level task of the task to be processed is abnormal, until the task level without abnormal tasks is determined, and the abnormal link analysis result is obtained.

[0011] In a possible design of the first aspect, obtaining the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed includes:

[0012] According to the identifier of the task to be processed, current running information of the task to be processed is obtained from a cache database, and historical running information of the task to be processed is obtained from a data warehouse;

[0013] The information in the cache database is obtained by consuming metadata information of each task in the big data platform; the data warehouse stores historical operation information of each task in the big data platform.

[0014] Optionally, before acquiring the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed, the method further includes:

[0015] Obtain metadata information for each task from the business system;

[0016] Clean the metadata information of each task and determine the current running information of each task;

[0017] The current running information of each task is stored in the cache database, and the running information of each task in the cache database has an expiration time.

[0018] Optionally, the method further includes:

[0019] Determining a periodic processing time of the cache database;

[0020] When the current time reaches the periodic processing time, the running information of at least one task whose storage time in the cache database exceeds the expiration time is transmitted to the data warehouse;

[0021] Update the historical running information of each task in the data warehouse.

[0022] In another possible design of the first aspect, determining whether the to-be-processed task is abnormal according to the current operation information, the historical operation information, and a pre-configured service level agreement includes:

[0023] Determining, based on the current operation information and the service level agreement, whether the completion time of the pending task exceeds the threshold time set in the service level agreement;

[0024] When the completion time of the task to be processed exceeds the threshold time set in the service level agreement, it is determined whether the task to be processed is abnormal according to the current operation information and the historical operation information.

[0025] In another possible design of the first aspect, based on the task level information, recursively analyzing whether the previous level task of the task to be processed is abnormal, until the task level without abnormal tasks is determined, and obtaining the abnormal link analysis result, includes:

[0026] According to the task level information, determine all previous level tasks of the task to be processed;

[0027] According to the running status of each task in the upper-level task, an abnormality analysis is performed on the tasks that meet the preset conditions to obtain an abnormal link analysis result.

[0028] Optionally, performing abnormal analysis on tasks that meet preset conditions according to the running status of each task in the previous level task to obtain abnormal link analysis results includes:

[0029] When there are unrunning tasks in the upper-level tasks, abnormal analysis is performed on all the unrunning tasks to obtain the abnormal link analysis result;

[0030] When there is no unrunning task in the previous level task, but there are running tasks and running completed tasks, abnormal analysis is performed on all running tasks and tasks whose running end time exceeds the threshold time to obtain the abnormal link analysis result;

[0031] When all tasks in the previous level are completed tasks, abnormal analysis is performed on the tasks in the previous level whose running end time exceeds the threshold time until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

[0032] In another possible design of the first aspect, the method further includes:

[0033] The abnormal link analysis result is output in the form of a link graph.

[0034] According to a second aspect of the present application, an embodiment of the present application provides an abnormal link analysis device, including:

[0035] A receiving module, configured to receive an abnormal link analysis request from a user, wherein the abnormal link analysis request includes: an identifier of a task to be processed;

[0036] An acquisition module, used for acquiring current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed;

[0037] A processing module is used to determine whether the task to be processed is abnormal based on the current operation information, the historical operation information and a pre-configured service level agreement, and when it is determined that the task to be processed is abnormal, recursively analyze whether the previous level task of the task to be processed is abnormal based on the task hierarchy information, until the task hierarchy of the task without abnormality is determined, thereby obtaining the abnormal link analysis result.

[0038] In a possible design of the second aspect, the acquisition module is specifically used to acquire the current running information of the task to be processed from the cache database according to the identifier of the task to be processed, and acquire the historical running information of the task to be processed from the data warehouse;

[0039] The information in the cache database is obtained by consuming metadata information of each task in the big data platform; the data warehouse stores historical operation information of each task in the big data platform.

[0040] Optionally, the acquisition module is further used to acquire metadata information of each task from the business system before acquiring the current running information and historical running information of the task to be processed according to the identifier of the task to be processed;

[0041] The processing module is also used to clean the metadata information of each task, determine the current running information of each task, and store the current running information of each task in the cache database. The running information of each task in the cache database has an expiration time.

[0042] Optionally, the processing module is further used to:

[0043] Determining a periodic processing time of the cache database;

[0044] When the current time reaches the periodic processing time, the running information of at least one task whose storage time in the cache database exceeds the expiration time is transmitted to the data warehouse;

[0045] Update the historical running information of each task in the data warehouse.

[0046] In another possible design of the second aspect, the processing module is used to determine whether the to-be-processed task is abnormal according to the current operation information, the historical operation information, and a pre-configured service level agreement, specifically:

[0047] The processing module is specifically used for:

[0048] Determining, based on the current operation information and the service level agreement, whether the completion time of the pending task exceeds the threshold time set in the service level agreement;

[0049] When the completion time of the task to be processed exceeds the threshold time set in the service level agreement, it is determined whether the task to be processed is abnormal according to the current operation information and the historical operation information.

[0050] In another possible design of the second aspect, the processing module is used to recursively analyze whether the previous level task of the task to be processed is abnormal based on the task level information, until the task level without abnormal tasks is determined, and the abnormal link analysis result is obtained, specifically:

[0051] The processing module is specifically used for:

[0052] According to the task level information, determine all previous level tasks of the task to be processed;

[0053] According to the running status of each task in the upper-level task, an abnormality analysis is performed on the tasks that meet the preset conditions to obtain an abnormal link analysis result.

[0054] Optionally, the processing module is used to perform an abnormality analysis on the tasks that meet the preset conditions according to the running status of each task in the previous level task, and obtain an abnormal link analysis result, specifically:

[0055] The processing module is specifically used for:

[0056] When there are unrunning tasks in the upper-level tasks, abnormal analysis is performed on all the unrunning tasks to obtain the abnormal link analysis result;

[0057] When there is no unrunning task in the previous level task, but there are running tasks and running completed tasks, abnormal analysis is performed on all running tasks and tasks whose running end time exceeds the threshold time to obtain the abnormal link analysis result;

[0058] When all tasks in the previous level are completed tasks, abnormal analysis is performed on the tasks in the previous level whose running end time exceeds the threshold time until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

[0059] In another possible design of the second aspect, the device further includes:

[0060] The output module is used to output the abnormal link analysis result in the form of a link diagram.

[0061] According to the third aspect of the present application, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect and various possible designs is implemented.

[0062] According to the fourth aspect of the present application, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the first aspect and various possible designs.

[0063] According to the fifth aspect of the present application, an embodiment of the present application provides a computer program product, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0064] The abnormal link analysis method, device, equipment and storage medium provided in the embodiment of the present application receive the abnormal link analysis request from the user, obtain the current operation information and historical operation information of the task to be processed according to the identification of the task to be processed included in the abnormal link analysis request, first determine whether the task to be processed is abnormal according to the current operation information, historical operation information and the pre-configured service level agreement, and when it is determined that the task to be processed is abnormal, recursively analyze whether the task of the previous level of the task to be processed is abnormal based on the task hierarchy information, until the task hierarchy of the task without abnormality is determined, and the abnormal link analysis result is obtained. In this technical solution, the analysis of abnormal links can be performed based on the needs of the user, and the cause of the abnormal link can be located, which improves the accuracy of task monitoring and provides the conditions for improving the data quality output by the big data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0066] Figure 1 A schematic diagram of a network architecture applicable to the abnormal link analysis method provided in an embodiment of the present application;

[0067] Figure 2 A flowchart of an abnormal link analysis method according to an embodiment of the present application;

[0068] Figure 3 A flowchart of a second embodiment of the abnormal link analysis method provided in the embodiment of the present application;

[0069] Figure 4 A flowchart of a third embodiment of the abnormal link analysis method provided in an embodiment of the present application;

[0070] Figure 5 A flowchart of a fourth embodiment of the abnormal link analysis method provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the distribution of task links at multiple levels in the task scheduling system in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of the structure of an abnormal link analysis device embodiment provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of the structure of an electronic device embodiment provided in an embodiment of the present application.

[0074] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] First, the terms involved in the embodiments of the present application are explained:

[0077] Scheduling system: The task scheduling system (Buffalo), also known as the workflow scheduling system, is a heavyweight product for offline computing on the big data platform. It not only carries out the synchronization work between various databases and data sets, but also carries out various offline data computing work. Through the task scheduling system, scheduled tasks can be managed conveniently and quickly, such as importing new data from the database to the data platform on a regular basis, and exporting data processed by the data platform to the database or file system. In addition, the task scheduling system also supports the establishment of dependencies between tasks, rapid addition and re-running of tasks, and powerful monitoring functions, thereby providing good job management services.

[0078] Data warehouse: Data warehouse (DW or DWH). A data warehouse is a strategic collection of all types of data support for decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes, and can provide guidance for business process improvement, monitoring time, cost, quality, and control for enterprises that require business intelligence.

[0079] Hive: Hive is a data warehouse tool based on Hadoop. It is used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files to a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution. The advantage of Hive is that it has a low learning cost and can achieve fast MapReduce statistics through similar SQL statements, making MapReduce simpler without having to develop a dedicated MapReduce application. Hive is very suitable for statistical analysis of data warehouses.

[0080] Kafka: Kafka is an open source stream processing platform written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action stream data of consumers in the website.

[0081] Redis: Remote dictionary server (Redis) is an open source, ANSI C-language, network-enabled, memory-based, or persistent log-based, Key-Value database that provides APIs in multiple languages.

[0082] Flink: Apache Flink is an open source stream processing framework, the core of which is a distributed streaming data flow engine written in Java and Scala. Flink executes any streaming data program in a data-parallel and pipelined manner. When Flink's pipeline is running, the system can execute batch and stream processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms.

[0083] SLA: Service level agreement (SLA) refers to an agreement or contract between a service provider and a customer on the quality, level, performance, etc. of the service, which is mutually recognized by both parties. A typical service level agreement includes the following: provisions of the parties involved in the services provided and the validity period of the agreement; time provisions during the service provision period, including testing, maintenance and upgrades; provisions for the number of users, locations and corresponding hardware services provided; description of the fault reporting process, including the conditions for escalating faults to a higher level of support; provisions for the expected response time for fault reports; description of the change request process. Optionally, the service level agreement may also include the expected time for completing routine change requests; provisions for service level objectives; service-related charging provisions; provisions for user responsibilities (user training, ensuring correct desktop configuration, no unnecessary software, no interference with the change management process, etc.); description of the process for resolving disagreements related to the service, etc.

[0084] With the rapid development of network technology and the growth of various corporate businesses, a large amount of unstructured data related to processes and rules has also exploded. The big data platform can provide services such as resource management, data calculation and analysis, data storage, service monitoring and management for massive data. Based on distributed file systems and distributed parallel computing, it integrates system installation, cluster configuration, security access control, monitoring and early warning and other services, providing reliable data support for corporate big data core business, business intelligence, operational analysis, business decision-making, etc.

[0085] In order to meet the user's requirements for the data quality output in the big data platform, the existing technology can monitor the operation of tasks running in the big data platform to ensure that the data can be produced on time and accurately.

[0086] It can be seen from the records in the above background technology that the task monitoring scheme in the related technology will trigger an alarm if the running time of the current monitored task does not meet the set time threshold, but it may not be caused by the abnormality of the task itself, but due to the abnormal operation of one or more parent tasks of the current task. It cannot directly locate the real cause of the task abnormality alarm, and there is a problem of low task monitoring accuracy.

[0087] In response to the above technical problems, the conceived process of the technical solution of the present application is as follows: through tracing the source, the task scheduling system of the big data platform is monitored, and the operation information of each task in the scheduling system is periodically processed, so that the operation information of each task in the current time period is stored separately from the operation information in the historical time period, which lays the foundation for the subsequent task exception analysis. Secondly, when an exception analysis request is received, the current task is first analyzed for exceptions. When it is determined that the current task is abnormal, the parent task of the current task is recursively analyzed until the last level of abnormal tasks or the level without abnormal tasks is determined. In this way, the real cause of the task abnormality can be determined, thereby improving the accuracy of task monitoring.

[0088] Based on the above technical conception process, the embodiment of the present application provides an abnormal link analysis method. When receiving the abnormal link analysis request from the user, the current operation information and historical operation information of the task to be processed can be obtained according to the identifier of the task to be processed included in the abnormal link analysis request. First, it is determined whether the task to be processed is abnormal based on the current operation information, historical operation information and the pre-configured service level agreement. When it is determined that the task to be processed is abnormal, based on the task hierarchy information, it is recursively analyzed whether the task above the task to be processed is abnormal, until the task hierarchy of the task without abnormality is determined, and the abnormal link analysis result is obtained. In this technical solution, the analysis of abnormal links can be performed based on the needs of the user, and the cause of the abnormal link can be located, which improves the accuracy of task monitoring and provides the conditions for improving the data quality output by the big data platform.

[0089] For example, Figure 1 A schematic diagram of a network architecture applicable to the abnormal link analysis method provided in the embodiment of the present application. Figure 1 As shown, in this embodiment, the network structure diagram may include: a data platform 11 and a link analysis device 12 connected to each other. The data platform 11 may include a data source 111, a data processing module 112 and a data storage module 113. The data storage module 113 is connected to the link analysis device 12, and the two realize information exchange.

[0090] The data provided by the data source 111 may be application data, data stored in a database, data logs, and data from other data sources, etc. The embodiment of the present application does not limit the source of the data in the data source 111, which can be determined based on the actual scenario and will not be elaborated here.

[0091] The data processing module 112 can perform pre-processing such as cleaning and classification on the data provided by the data source 111 , and store the processed data in the data storage module 113 .

[0092] The data storage module 113 may include: a cache database 1131 and a data warehouse 1132. The data in the cache database 1131 may be the current operation information obtained by the data platform 11 executing each task in the current time period, and the data in the data warehouse 1132 may be the expired data in the cache database 1131, which may be used as the historical operation information of each task in the cache database 1131.

[0093] In one embodiment of the present application, when the link analysis device 12 has a human-computer interaction interface, the user can send a link analysis request including a pending task identifier to the link analysis device 12 through the human-computer interaction interface, so that the link analysis device 12 can obtain the current operation information and historical operation information of the pending task from the cache database 1131 and the data warehouse 1132 included in the data storage module 113, respectively. Accordingly, the link analysis device 12 can perform analysis and positioning of abnormal tasks based on the current operation information, historical operation information and pre-configured service level agreement of the pending task. Optionally, the link analysis device 12 can also output abnormal link analysis results through the human-computer interaction interface.

[0094] In another embodiment of the present application, the network structure diagram may also include: a user terminal 13. In this case, the user may send an abnormal link analysis request to the link analysis device 12 through the user terminal 13, so that based on the identifier of the task to be processed in the abnormal link analysis request, the current operation information and historical operation information of the task to be processed are obtained from the cache database 1131 and the data warehouse 1132 included in the data storage module 113, respectively. Accordingly, the link analysis device 12 may perform analysis and positioning of the abnormal task based on the current operation information, historical operation information and pre-configured service level agreement of the task to be processed. Exemplarily, the link analysis device 12 may feed back the abnormal link analysis results to the user terminal 13, so that the user terminal 13 performs the display of the abnormal link analysis results.

[0095] Optionally, in another embodiment of the present application, the network structure diagram may further include: a display device 14, which may receive and display the abnormal link analysis results sent by the link analysis device 12.

[0096] Optionally, in an embodiment of the present application, the link analysis device may be implemented through a server or a terminal device, which is not limited here.

[0097] It is understandable that the embodiment of the present application does not limit the type and quantity of devices included in the network structure diagram, which can be set according to scenario requirements and will not be elaborated here.

[0098] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0099] Figure 2 The flowchart of the abnormal link analysis method embodiment 1 provided in the present application is as follows. Figure 1 The link analysis device in the network architecture diagram is explained as an execution subject. Figure 2 As shown, the abnormal link analysis method may include the following steps:

[0100] S201. Receive an abnormal link analysis request from a user, where the abnormal link analysis request includes: an identifier of a task to be processed.

[0101] In the embodiments of the present application, the link analysis device can be abstracted into a monitoring system, but the difference between it and the monitoring system in the prior art is that the link analysis device has an interface capable of receiving external requests.

[0102] Exemplarily, the link analysis device can start an http service by running a loaded program, and through the http service can receive abnormal link analysis requests submitted by users for tasks to be processed, so that the link analysis device can analyze the running status of each task in the task scheduling system in the big data platform.

[0103] Exemplarily, after the http service of the link analysis device receives an abnormal link analysis request from a user, it parses the abnormal link analysis request to obtain an identifier of the task to be processed.

[0104] In the embodiment of the present application, the abnormal link analysis request is mainly for the analysis of abnormal links. In practical applications, the abnormal link analysis request can also be replaced by the latest link analysis request, so that the link analysis device performs link analysis based on the latest link analysis request. It is understandable that the link analysis device can use different analysis logics for different link analysis requests, which will not be elaborated here.

[0105] S202: According to the identifier of the task to be processed, obtain the current running information and the historical running information of the task to be processed.

[0106] In an embodiment of the present application, the link analysis device parses the abnormal link analysis request and, after obtaining the identification of the task to be processed, can query the data storage module used to store the running information of the task to be processed based on the identification of the task to be processed to obtain the current running information and historical running information of the task to be processed.

[0107] Exemplarily, in one embodiment of the present application, the link analysis device may obtain the current running information of the task to be processed from the cache database according to the identifier of the task to be processed, and obtain the historical running information of the task to be processed from the data warehouse.

[0108] Among them, the information in the cache database is obtained by consuming the metadata information of each task in the big data platform; the data warehouse stores the historical operation information of each task in the big data platform.

[0109] It can be understood that the cache database and data warehouse can be components of the data storage module in the big data platform, or components of the link analysis device. Their specific locations can be set according to actual scenarios, and this embodiment does not limit them.

[0110] In this step, by consuming the metadata information of each task in the big data platform, the current running information of each task can be obtained and stored in the cache database for subsequent use. Optionally, the data of each task in the cache database has an expiration time. When the storage time of the data of each task in the cache database exceeds the expiration time, the historical running information of each task is obtained. Optionally, the historical running information of each task is periodically updated to the data warehouse for subsequent acquisition.

[0111] In practical applications, since the implementation of each business request usually involves multiple tasks to be executed in sequence, the execution of each task is interrelated. Connecting the information of these tasks in series forms a link. The link analysis described in this application is to analyze the link where the task to be processed is located.

[0112] S203: Determine whether the task to be processed is abnormal based on the current operation information, historical operation information and a pre-configured service level agreement.

[0113] In an embodiment of the present application, a service level agreement pre-configured for each task is stored in the link analysis device, and the service level agreement includes a running time threshold of the corresponding task, etc. Under normal circumstances, the running time threshold of the corresponding task included in the service level agreement can be used as an important basis for judging whether the task is abnormal. Therefore, when a user requests to view an abnormal link, the task to be processed can be compared to see whether it is abnormal based on the parameter information passed in by the user and the historical situation of the task running. Specifically, after obtaining the current running information and historical running information of the task to be processed, the time threshold of the task to be processed can be determined based on the pre-configured service level agreement, and then the current running information and the historical running information are combined to jointly judge whether the task to be processed is abnormal, so as to improve the accuracy of the judgment.

[0114] S204: When it is determined that the task to be processed is abnormal, based on the task level information, recursively analyze whether the task at the previous level of the task to be processed is abnormal, until the task level without abnormal tasks is determined, and the abnormal link analysis result is obtained.

[0115] In this embodiment, each task may involve multiple links, and each link may include multiple levels. When determining that the task to be processed is abnormal, in order to determine the real reason for the abnormality of the task to be processed, it can be analyzed upwards by recursive analysis based on the task level information, that is, first analyze whether the task above the task to be processed is abnormal, and when the task above is also abnormal, analyze upwards until the task level without abnormal tasks is analyzed, that is, the root cause of the task to be processed is determined, thereby obtaining the abnormal link analysis result corresponding to the abnormal link analysis request.

[0116] The abnormal link analysis method provided in the embodiment of the present application receives an abnormal link analysis request from a user, obtains the current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed included in the abnormal link analysis request, and then determines whether the task to be processed is abnormal according to the current operation information, the historical operation information and a pre-configured service level agreement. When it is determined that the task to be processed is abnormal, based on the task hierarchy information, recursively analyzes whether the previous level task of the task to be processed is abnormal, until the task hierarchy of the task without abnormality is determined, and the abnormal link analysis result is obtained. This scheme can locate the cause of the abnormal link, improve the accuracy of task monitoring, and provide implementation conditions for improving the data quality output by the big data platform.

[0117] Based on the above embodiments, Figure 3 The flowchart of the abnormal link analysis method embodiment 2 provided in the present application is as follows. Figure 3 As shown, in this embodiment, before the above S202, the abnormal link analysis method may further include the following steps:

[0118] S301. Obtain metadata information of each task from the business system.

[0119] Exemplarily, in an embodiment of the present application, the metadata information of each task can be generated in real time during the process of scheduling the task operation by the task scheduling system. Optionally, in a big data platform, a task has idle (i.e., successful), failed, running, etc. states during operation. When the state of the task changes, metadata information of the task is generated, which can be written into the business system of the big data platform. Correspondingly, in an embodiment of the present application, the link analysis device can consume the metadata information of each task in the business system through the real-time computing engine, thereby obtaining the metadata information of each task for analysis.

[0120] Optionally, the business system may also be a distributed publish-subscribe messaging system, such as Kafka, which can process all action stream data of users on the website. The real-time computing engine may be flink. In practical applications, the real-time computing engine may also be a computing engine such as storm or spark. This embodiment does not limit the specific implementation of the real-time computing engine.

[0121] Optionally, the metadata information of each task may include task name, task ID, task running status, task start running time, task running end time, etc. This embodiment does not limit the specific content of each metadata information, which can be determined according to the actual scenario.

[0122] S302: Clean the metadata information of each task to determine the current running information of each task.

[0123] Exemplarily, in this embodiment, in order to avoid storing too much data that is useless for link analysis in the cache database, the link analysis device can first clean and filter the metadata information of each task, eliminate some data that is useless for exception analysis, and obtain the current operation information of each task.

[0124] S303: Store the current running information of each task in a cache database. The running information of each task in the cache database has an expiration time.

[0125] Optionally, the link analysis device can save the current running information of each task obtained into a cache database and set an expiration time for the running information of each task. Exemplarily, the link analysis device can set a 24-hour expiration time for the running information of each task, that is, cache the running information of each task for one day.

[0126] Exemplarily, the cache database may be any one of Redis, Hbase, ClickHouse and the like, which is not limited in this embodiment.

[0127] It is understandable that when performing abnormal link analysis on the task to be processed, the running status of the superior task (parent task) needs to be utilized. Therefore, the link analysis device also needs to cache the running information of the parent task in the cache database in advance. The storage method is similar to that of the current task, which will not be elaborated here.

[0128] S304: Determine the periodic processing time of the cache database.

[0129] In the embodiment of the present application, the cache database has a periodic processing time, so that the link analysis device can periodically process the data in the cache database based on the periodic processing time.

[0130] For example, if the processing cycle of the cache database is 1 day, the cache database can be updated at the same time every day according to the scheduling date of the task, for example, at 11:55 pm every day, so that it can be used when performing real-time analysis the next day. It is understandable that the embodiments of the present application do not limit the specific cycle processing time, which can be determined according to the actual scenario.

[0131] S305: When the current time reaches the periodic processing time, the running information of at least one task whose storage time in the cache database exceeds the expiration time is transmitted to the data warehouse.

[0132] In an embodiment of the present application, the link analysis device may compare the current moment with the above-mentioned periodic processing moment, and process the operation of each task in the cache database at the periodic processing moment of each cycle. For example, the operation information of at least one task in the cache database whose storage time exceeds the expiration time is transmitted to the data warehouse, so that the link analysis device can use the operation information in the data warehouse in the next cycle, that is, the historical operation information of the task.

[0133] S306: Update the historical operation information of each task in the data warehouse.

[0134] Optionally, after receiving the running information of at least one task transmitted from the cache database, the link analysis device may periodically update the historical running information of each task.

[0135] Exemplarily, the historical operation information of each task in the data warehouse may include, for example, the average start operation time, the average end operation time, the average execution time, the previous and next level information (task lineage information), the tag information of the task, etc. The embodiment of the present application does not limit the specific embodiment of the historical operation information, which can be determined according to the actual scenario.

[0136] The abnormal link analysis method provided by the embodiment of the present application obtains the metadata information of each task from the business system, cleans the metadata information of each task, determines the current operation information of each task, and stores the current operation information of each task in the cache database. The operation information of each task in the cache database has an expiration time; accordingly, when the current moment reaches the periodic processing moment of the cache database, the operation information of at least one task whose storage time in the cache database exceeds the expiration time can be transmitted to the data warehouse, and the historical operation information of each task in the data warehouse is updated. This technical solution lays the foundation for real-time link analysis of subsequent tasks by storing the current operation information of each task in the cache database and updating each historical operation information to the data warehouse, thereby improving the efficiency and accuracy of abnormal location.

[0137] Based on the above embodiments, Figure 4 The flowchart of the abnormal link analysis method embodiment 3 provided in the present application is as follows. Figure 4 As shown, in this embodiment, the above S203 can be implemented by the following steps:

[0138] S401. Determine, based on current operation information and a service level agreement, whether the completion time of the task to be processed exceeds a threshold time set in the service level agreement.

[0139] Exemplarily, a preconfigured service level agreement is stored in the link analysis device, and each service level agreement includes a threshold time for the running of each task. Therefore, first, based on the identifier of the task to be processed, the service level agreement of the task to be processed can be queried to obtain the threshold time set in the service level agreement, and then the completion time of the task to be processed can be obtained based on the current running information of the task to be processed, and the two can be compared to determine whether the completion time of the task to be processed exceeds the threshold time set in the service level agreement.

[0140] As an example, when the completion time of the pending task does not exceed the threshold time set in the service level agreement, it indicates that the pending task is normal and does not need to be recursively analyzed upwards.

[0141] As another example, when the completion time of a pending task exceeds a threshold time set in a service level agreement, it is necessary to determine whether the pending task is abnormal.

[0142] S402: When the completion time of the pending task exceeds the threshold time set in the service level agreement, determine whether the pending task is abnormal based on the current operation information and the historical operation information.

[0143] Optionally, when the completion time of the pending task exceeds the threshold time set in the service level agreement, the current running information of the pending task can be compared with the historical running information of the pending task, for example, the start running time in the current running information is compared with the start running time in the historical running information, or the end running time in the current running information is compared with the end running time in the historical running information, or the relationship between the execution time of the pending task and the average execution time is directly judged to determine whether the pending task is abnormal.

[0144] For example, when the error between the execution time of the task to be processed and the average execution time is less than the preset error, the task to be processed can be considered normal, and no upward recursive analysis is required. When the error between the execution time of the task to be processed and the average execution time is greater than or equal to the preset error, the task to be processed can be considered abnormal, and an upward recursive analysis is required to determine the real reason for the abnormality of the task to be processed.

[0145] The abnormal link analysis method provided in the embodiment of the present application determines whether the completion time of the pending task exceeds the threshold time set in the service level agreement based on the current operation information and the service level agreement. When the completion time of the pending task exceeds the threshold time set in the service level agreement, it determines whether the pending task is abnormal based on the current operation information and historical operation information. In this technical solution, whether the pending task is abnormal is determined based on the current operation information, historical operation information and the service level agreement, which improves the accuracy of abnormal analysis, thereby laying a foundation for improving the quality of data output by the big data platform.

[0146] Based on the above embodiments, Figure 5 This is a flow chart of the fourth embodiment of the abnormal link analysis method provided in the present application. Figure 5 As shown, in this embodiment, the above S204 can be implemented by the following steps:

[0147] S501. Determine all upper-level tasks of the task to be processed according to the task level information.

[0148] In an embodiment of the present application, tasks in a task scheduling system in a big data platform usually have multiple levels, and each task can be on the same link with tasks of different levels. In order to accurately analyze the cause of the abnormality of the task to be processed, it is necessary to determine all the previous level tasks of the task to be processed based on the task hierarchy information in the task scheduling system.

[0149] For example, Figure 6 FIG. 1 is a schematic diagram of the distribution of multiple levels of task links in the task scheduling system in the embodiment of the present application. Figure 6As shown, in this embodiment, the task scheduling system is explained as having 8 levels, which are level 1 to level 8 from top to bottom. Each level can have the same or different number of tasks, for example, level 1 has 3 tasks, namely task 11, task 12 and task 13, and level 2 has 5 tasks, namely task 21 to task 25.

[0150] Optional, in Figure 6 In the schematic diagram shown, there can be multiple upper-level tasks (i.e., parent tasks) at each level, and they can be located at different levels. Figure 6 In the schematic diagram shown, task 81 at the 8th level has four upper-level tasks, one of which is located at the 7th level, namely task 71, and the other three upper-level tasks are all located at the 2nd level, namely task 23, task 24 and task 25.

[0151] S502: According to the running status of each task in the previous level task, perform abnormal analysis on the tasks that meet the preset conditions to obtain abnormal link analysis results.

[0152] In an embodiment of the present application, the link analysis device can determine the previous-level task that needs to be recursively analyzed based on the running status of each task in the previous-level task, and in the recursive analysis process, determine the real cause of the abnormality of the task to be processed to obtain the abnormal link analysis result.

[0153] Exemplarily, this step can be implemented by the following scheme:

[0154] As an example, when there are unexecuted tasks in the previous level tasks, abnormality analysis is performed on all the unexecuted tasks to obtain abnormal link analysis results.

[0155] If there are unrunning tasks in the previous level tasks, it indicates that the previous level tasks may not be run due to an exception. At this time, you only need to analyze all the unrunning tasks. By performing an exception analysis on the unrunning previous level tasks, you can determine the real cause of the exception of the task to be processed, and thus obtain the abnormal link analysis results.

[0156] As another example, when there are no unrunning tasks in the previous level tasks, but there are running tasks and completed tasks, an abnormal analysis is performed on all running tasks and tasks whose running end time exceeds the threshold time to obtain abnormal link analysis results.

[0157] Optionally, if there are no unrunning tasks in the previous level tasks, it means that the previous level tasks can all start execution normally. However, since there are running tasks and completed tasks, all running tasks and tasks whose running end time exceeds the threshold time can be analyzed for abnormalities. The real reason why the previous level task is running and the reason why the running end time exceeds the threshold time can be analyzed, and the abnormal link analysis results can be obtained.

[0158] As another example, when all tasks in the previous level are completed tasks, abnormal analysis is performed on the tasks in the previous level whose running end time exceeds the threshold time until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

[0159] Optionally, if all tasks in the previous level task are completed tasks, it means that all previous level tasks can be completed. At this time, an abnormal analysis can be performed on the tasks whose running end time exceeds the threshold time to analyze whether the previous level task is abnormal. If there are abnormal tasks in the previous level task whose running end time exceeds the threshold time, the abnormal situation of the parent task of the previous level task is recursively analyzed upward until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

[0160] The abnormal link analysis method of the embodiment of the present application determines all the upper-level tasks of the task to be processed according to the task hierarchy information, and performs abnormal analysis on the tasks that meet the preset conditions according to the running status of each task in the upper-level task to obtain the abnormal link analysis result. In this technical solution, the real cause of the link abnormality can be determined through the recursive analysis method, which improves the task monitoring efficiency and monitoring effect in the big data platform.

[0161] Furthermore, in an embodiment of the present application, the abnormal link analysis method may further include the following steps:

[0162] The abnormal link analysis results are output in the form of a link graph.

[0163] In this embodiment, after obtaining the abnormal analysis result, the link analysis device may represent it in the form of a link graph, and finally output it in the form of a link graph.

[0164] As an example, when the link analysis device has a display function, the link analysis device can output the abnormal link analysis result in the form of a link diagram through its own display function.

[0165] As another example, when the link analysis device does not have a display function, the link analysis device can transmit the abnormal link analysis results in the form of a link graph to a display device connected thereto so that the display device can display them.

[0166] It can be understood that both of the above examples can output the abnormal link analysis results to the user in the form of a link diagram, that is, whenever the user queries the link anomaly based on the task to be processed, a real-time abnormal link analysis result will be obtained and displayed to the user, so that the user can obtain an intuitive analysis result.

[0167] For example, refer to the above Figure 6 As shown, there may be abnormal tasks in tasks at each level, for example, task 11 in level 1, task 23 in level 2, task 31 in level 3, task 43 in level 4, tasks 53 and 54 in level 5, and task 61 in level 6. It can be understood that Figure 6 The link diagram shown is the result of real-time analysis based on each abnormal link analysis request. The link connecting each abnormal task to the next subtask is an abnormal link. Figure 6 The thin solid line in the figure is a normal link line, and the thick dotted line is an abnormal link line. The arrow of each link line represents the task from the upper level to the lower level. This embodiment does not limit it.

[0168] From the above analysis, it can be seen that the embodiment of the present application obtains the current running information of the task to be processed, such as the task status, running start time, running end time, running duration and other information, and then combines the historical running status of the task and the direct blood relationship (hierarchical relationship) of the task to perform real-time analysis of the abnormal link, and can obtain the abnormal link analysis results, and display the abnormal link analysis results in the form of a link diagram, so that when the user performs abnormal link analysis, the real situation of the cause of the abnormality can be located, and by intuitively observing the running of the upstream task, it can be judged whether the task abnormality is caused by the parent task or the abnormal execution of the task itself, thereby improving the accuracy of task monitoring.

[0169] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0170] Figure 7 This is a schematic diagram of the structure of an abnormal link analysis device embodiment provided in the present application. Figure 7 As shown, the abnormal link analysis device may include:

[0171] The receiving module 701 is used to receive a user's abnormal link analysis request, wherein the abnormal link analysis request includes: an identifier of a task to be processed;

[0172] The acquisition module 702 is used to acquire the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed;

[0173] Processing module 703 is used to determine whether the task to be processed is abnormal based on the current operation information, the historical operation information and the pre-configured service level agreement, and when it is determined that the task to be processed is abnormal, recursively analyze whether the previous level task of the task to be processed is abnormal based on the task hierarchy information, until the task hierarchy of the task without abnormality is determined, and the abnormal link analysis result is obtained.

[0174] In a possible design of an embodiment of the present application, the acquisition module 702 is specifically used to acquire the current running information of the task to be processed from the cache database according to the identifier of the task to be processed, and acquire the historical running information of the task to be processed from the data warehouse;

[0175] The information in the cache database is obtained by consuming metadata information of each task in the big data platform; the data warehouse stores historical operation information of each task in the big data platform.

[0176] Optionally, the acquisition module 702 is further configured to acquire metadata information of each task from a business system before acquiring the current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed;

[0177] The processing module 703 is also used to clean the metadata information of each task, determine the current running information of each task, and store the current running information of each task in the cache database. The running information of each task in the cache database has an expiration time.

[0178] Optionally, the processing module 703 is further used to:

[0179] Determining a periodic processing time of the cache database;

[0180] When the current time reaches the periodic processing time, the running information of at least one task whose storage time in the cache database exceeds the expiration time is transmitted to the data warehouse;

[0181] Update the historical running information of each task in the data warehouse.

[0182] In another possible design of the embodiment of the present application, the processing module 703 is used to determine whether the to-be-processed task is abnormal according to the current operation information, the historical operation information and the pre-configured service level agreement, specifically:

[0183] The processing module 703 is specifically used for:

[0184] Determining, based on the current operation information and the service level agreement, whether the completion time of the pending task exceeds the threshold time set in the service level agreement;

[0185] When the completion time of the task to be processed exceeds the threshold time set in the service level agreement, it is determined whether the task to be processed is abnormal according to the current operation information and the historical operation information.

[0186] In another possible design of the embodiment of the present application, the processing module 703 is used to recursively analyze whether the previous level task of the task to be processed is abnormal based on the task level information, until the task level without abnormal tasks is determined, and the abnormal link analysis result is obtained, which is specifically:

[0187] The processing module 703 is specifically used for:

[0188] According to the task level information, determine all previous level tasks of the task to be processed;

[0189] According to the running status of each task in the upper-level task, an abnormality analysis is performed on the tasks that meet the preset conditions to obtain an abnormal link analysis result.

[0190] Optionally, the processing module 703 is used to perform an abnormality analysis on the tasks that meet the preset conditions according to the running status of each task in the previous level task, and obtain an abnormal link analysis result, specifically:

[0191] The processing module 703 is specifically used for:

[0192] When there are unrunning tasks in the upper-level tasks, abnormal analysis is performed on all the unrunning tasks to obtain the abnormal link analysis result;

[0193] When there is no unrunning task in the previous level task, but there are running tasks and running completed tasks, abnormal analysis is performed on all running tasks and tasks whose running end time exceeds the threshold time to obtain the abnormal link analysis result;

[0194] When all tasks in the previous level are completed tasks, abnormal analysis is performed on the tasks in the previous level whose running end time exceeds the threshold time until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

[0195] In another possible design of the embodiment of the present application, the device further includes:

[0196] The output module 704 is used to output the abnormal link analysis result in the form of a link graph.

[0197] The device provided in the embodiment of the present application can be used to execute the technical solution described in the method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0198] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0200] Figure 8 This is a schematic diagram of the structure of an electronic device embodiment provided in the present application. Figure 8 As shown, the electronic device may include: a processor 801, a memory 802, a communication interface 803 and a system bus 804, wherein the memory 802 and the communication interface 803 are connected to the processor 801 through the system bus 804 and communicate with each other, the memory 802 is used to store computer instructions, the communication interface 803 is used to communicate with other devices, and the processor 801 implements the technical solution described in the above method embodiment when executing the computer instructions.

[0201] Optionally, in an embodiment of the present application, the electronic device may further include a user operation interface 805, and the user operation interface 805 may be used for a user's abnormal link analysis request.

[0202] Among them, the above Figure 7 The receiving module 701 and the output module 704 in the abnormal link analysis device shown can be implemented through the communication interface 803 , and the above-mentioned processing module 703 can be implemented through the processor 801 .

[0203] In this Figure 8 In the embodiment, the processor 801 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0204] The memory 802 may include a random access memory (RAM), may include a read-only memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0205] The communication interface 803 is used to implement the communication between the database access device and other devices (such as the client, the read-write library and the read-only library).

[0206] The system bus 804 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0207] Optionally, an embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution described in the above method embodiment.

[0208] Optionally, an embodiment of the present application further provides a chip for running instructions, wherein the chip is used to execute the technical solution described in the above method embodiment.

[0209] An embodiment of the present application also provides a computer program product, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0210] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the application disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0211] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for analyzing abnormal links, characterized in that: include: Receive a user's abnormal link analysis request, the abnormal link analysis request including: an identifier of a task to be processed; According to the identifier of the task to be processed, obtaining the current running information and the historical running information of the task to be processed; Determining whether the task to be processed is abnormal according to the current operation information, the historical operation information and a pre-configured service level agreement; When it is determined that the task to be processed is abnormal, all upper-level tasks of the task to be processed are determined according to the task level information; When there are unrunning tasks in the upper-level tasks, abnormal analysis is performed on all the unrunning tasks to obtain the abnormal link analysis result; When there is no unrunning task in the previous level task, but there are running tasks and running completed tasks, abnormal analysis is performed on all running tasks and tasks whose running end time exceeds the threshold time to obtain the abnormal link analysis result; When all tasks in the previous level are completed tasks, abnormal analysis is performed on the tasks in the previous level whose running end time exceeds the threshold time until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

2. The method according to claim 1, characterized in that The step of obtaining the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed includes: According to the identifier of the task to be processed, current running information of the task to be processed is obtained from a cache database, and historical running information of the task to be processed is obtained from a data warehouse; The information in the cache database is obtained by consuming metadata information of each task in the big data platform; the data warehouse stores historical operation information of each task in the big data platform.

3. The method according to claim 2, characterized in that Before acquiring the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed, the method further includes: Obtain metadata information for each task from the business system; Clean the metadata information of each task and determine the current running information of each task; The current running information of each task is stored in the cache database, and the running information of each task in the cache database has an expiration time.

4. The method according to claim 3, characterized in that: The method further comprises: Determining a periodic processing time of the cache database; When the current time reaches the periodic processing time, the running information of at least one task whose storage time in the cache database exceeds the expiration time is transmitted to the data warehouse; Update the historical running information of each task in the data warehouse.

5. The method according to claim 1, characterized in that The determining whether the to-be-processed task is abnormal according to the current operation information, the historical operation information and a pre-configured service level agreement includes: Determining, based on the current operation information and the service level agreement, whether the completion time of the pending task exceeds a threshold time set in the service level agreement; When the completion time of the task to be processed exceeds the threshold time set in the service level agreement, it is determined whether the task to be processed is abnormal according to the current operation information and the historical operation information.

6. The method according to claim 1, characterized in that The method further comprises: The abnormal link analysis result is output in the form of a link graph.

7. An abnormal link analysis device, characterized in that: include: A receiving module, configured to receive an abnormal link analysis request from a user, wherein the abnormal link analysis request includes: an identifier of a task to be processed; An acquisition module, used for acquiring current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed; A processing module, configured to determine whether the task to be processed is abnormal according to the current operation information, the historical operation information and a pre-configured service level agreement, and when it is determined that the task to be processed is abnormal, recursively analyze whether the task at the previous level of the task to be processed is abnormal based on the task level information, until the task level of the task without abnormality is determined, thereby obtaining an abnormal link analysis result; The processing module is specifically used to determine all upper-level tasks of the task to be processed according to the task level information when it is determined that the task to be processed is abnormal; When there are unrunning tasks in the upper-level tasks, abnormal analysis is performed on all the unrunning tasks to obtain the abnormal link analysis result; When there is no unrunning task in the previous level task, but there are running tasks and running completed tasks, abnormal analysis is performed on all running tasks and tasks whose running end time exceeds the threshold time to obtain the abnormal link analysis result; When all tasks in the previous level are completed tasks, abnormal analysis is performed on the tasks in the previous level whose running end time exceeds the threshold time until the task level without abnormal tasks is determined to obtain the abnormal link analysis result.

8. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product comprising: A computer program, characterized in that when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 6.

Images