Abnormal task detection method, device, equipment and computer storage medium

Abstract

The present invention relates to the technical field of financial technology (Fintech) and discloses a method for detecting abnormal tasks, the method comprising: analyzing multiple tasks in a scheduling platform to determine the association relationships between each of the tasks and multiple tables in the scheduling platform; determining the dependency relationships between the tasks based on each of the association relationships, and detecting each of the tasks based on each of the dependency relationships to determine whether there are any abnormal tasks among the tasks; if so, treating the abnormal tasks as abnormal tasks. The present invention also discloses an abnormal task detection device, equipment, and a computer storage medium. The present invention improves the accuracy of abnormal task detection in the scheduling platform.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to an abnormal task detection method, device, equipment and computer storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (such as big data, distributed systems, blockchain, and artificial intelligence) are being applied in the financial sector. Traditional finance is gradually shifting towards financial technology (Fintech). However, the security and real-time requirements of the financial industry also place higher demands on technology. For example, existing scheduling platforms that use big data are generally focused on functional development. Although relational databases are used to detect closed-loop tasks in abnormal tasks, their detection efficiency is low. Moreover, relational database-based methods are not suitable for more analytical work and cannot accurately detect other types of abnormal tasks. Therefore, how to improve the accuracy of abnormal task detection in scheduling platforms has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The main purpose of the present invention is to propose an abnormal task detection method, device, equipment and computer storage medium, aiming to improve the accuracy of abnormal task detection in the scheduling platform.

[0004] To achieve the above object, the present invention provides a method for detecting abnormal tasks, which comprises the following steps:

[0005] Analyzing multiple tasks in the scheduling platform to determine the association relationship between each task and multiple tables in the scheduling platform;

[0006] Determining dependencies between the tasks based on the association relationships, and detecting the tasks based on the dependencies to determine whether there is an abnormal task among the tasks;

[0007] If so, the task with the exception is treated as an abnormal task.

[0008] Optionally, the step of detecting each of the tasks based on each of the dependency relationships includes:

[0009] Calculating the strongly connected components corresponding to each of the tasks based on each of the dependency relationships, and detecting whether there is a closed-loop task in each of the tasks based on the strongly connected components;

[0010] If a closed-loop task exists, the closed-loop task is regarded as a task with an exception.

[0011] Optionally, the step of detecting each of the tasks based on each of the dependency relationships further includes:

[0012] Obtaining a configuration dependency relationship preset in a task configuration information table corresponding to the scheduling platform;

[0013] Traversing each of the tasks in sequence, determining a current dependency of the currently traversed task based on each of the dependency relationships, and matching the current dependency with the configuration dependency;

[0014] If there is no match, the task currently being traversed is treated as a task with an exception until the traversal of each task is completed.

[0015] Optionally, the step of detecting each of the tasks based on each of the dependency relationships further includes:

[0016] Detecting each of the tasks in sequence based on each of the dependency relationships, parsing a target task in each of the tasks into a syntax tree using a syntax parser, and determining whether the level of the syntax tree is greater than a preset level;

[0017] If it is greater, the target task is treated as an abnormal task.

[0018] Optionally, the step of detecting each of the tasks based on each of the dependency relationships further includes:

[0019] Collecting the task attributes of each of the tasks in sequence based on each of the dependency relationships, traversing each of the task attributes in sequence, performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes, and detecting whether the key information meets preset requirements;

[0020] If it does not meet the requirements, the task corresponding to the currently traversed task attribute will be treated as an abnormal task until the traversal of the task attributes is completed.

[0021] Optionally, the step of performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes includes:

[0022] Perform weighted word segmentation on the currently traversed task attributes to obtain multiple word segmentations corresponding to the currently traversed task attributes, and detect whether any word in each of the word segmentations is a noun;

[0023] If it exists, the weight corresponding to the noun is increased, a preset number of target weights are obtained according to the size of the weights corresponding to each of the participles, and the participles corresponding to each of the target weights are used as the key information of the currently traversed task attributes.

[0024] Optionally, the step of analyzing the plurality of tasks in the scheduling platform to determine the association relationship between each of the tasks and the plurality of tables in the scheduling platform includes:

[0025] Analyze multiple tasks in the scheduling platform to obtain table-level kinship relationships between each task and multiple tables in the scheduling platform;

[0026] The table-level blood relationships are disassembled in a preset form to determine the association relationships between each of the tasks and the multiple tables in the scheduling platform.

[0027] In addition, to achieve the above-mentioned purpose, the present invention further provides an abnormal task detection device, the abnormal task detection device comprising:

[0028] A determination module, configured to analyze multiple tasks in the scheduling platform to determine associations between each of the tasks and multiple tables in the scheduling platform;

[0029] a detection module, configured to determine the dependency relationships between the tasks based on the association relationships, and detect the tasks based on the dependency relationships to determine whether there is any abnormal task among the tasks;

[0030] A module exists, for treating the task with the exception as an abnormal task if the task exists.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides an abnormal task detection device, which includes: a memory, a processor, and an abnormal task detection program stored on the memory and capable of running on the processor. When the abnormal task detection program is executed by the processor, the steps of the abnormal task detection method described above are implemented.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which an abnormal task detection program is stored. When the abnormal task detection program is executed by a processor, the steps of the abnormal task detection method described above are implemented.

[0033] The present invention analyzes multiple tasks in a scheduling platform to determine the associations between each task and multiple tables in the scheduling platform; determines the dependencies between each task based on the associations, and detects each task based on the dependencies to determine whether there are any tasks with abnormalities in each task; if so, the tasks with abnormalities are treated as abnormal tasks. By analyzing multiple tasks in the scheduling platform, determining the dependencies between each task, and then detecting abnormal tasks based on the dependencies, the efficiency of detecting abnormal tasks is improved. Since it is constructed based on the relationship between tables and tasks, it also ensures that the accuracy of abnormal task detection in the scheduling platform is improved without affecting the normal operation of the tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0035] Figure 2 This is a flow chart of the first embodiment of the abnormal task detection method of the present invention;

[0036] Figure 3 Schematic diagram of the device module of the abnormal task detection device of the present invention;

[0037] Figure 4 Schematic diagram of the process of task analysis in the abnormal task detection method of the present invention;

[0038] Figure 5 A schematic diagram of a process for determining dependencies between construction tasks in the abnormal task detection method of the present invention;

[0039] Figure 6 This is a diagram showing closed-loop tasks in the abnormal task detection method of the present invention;

[0040] Figure 7 Schematic diagram of over-matching and missing task detection in the abnormal task detection method of the present invention;

[0041] Figure 8 Schematic diagram of the task syntax detection process in the abnormal task detection method of the present invention;

[0042] Figure 9 This is a schematic diagram of the task weight segmentation process in the abnormal task detection method of the present invention.

[0043] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0046] The abnormal task detection device in the embodiment of the present invention may be a PC or a server device.

[0047] like Figure 1 As shown, the abnormal task detection device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an abnormal task detection program.

[0050] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the abnormal task detection program stored in the memory 1005 and perform the operations in the following abnormal task detection method.

[0051] Based on the above hardware structure, an embodiment of the abnormal task detection method of the present invention is proposed.

[0052] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a method for detecting abnormal tasks according to the present invention. The method includes:

[0053] Step S10, analyzing multiple tasks in the scheduling platform to determine the association relationship between each task and multiple tables in the scheduling platform;

[0054] In this embodiment, by performing lineage analysis on the logs of each task in the scheduling platform, a first relationship centered on the task is obtained. The final form of this first relationship can be (source library, source table)-[relationship: task]-(target library, target table). This first relationship is then stored in the entity-relationship format of the Neo4J graph database. The table-level relationship is then decomposed according to a preset format, such as (task)-[relationship: library, table]-(task), ultimately obtaining two task-level relationships. Specifically, the relationship after decomposing (source library, source table)-[relationship: task]-(target library, target table) is ()-[relationship: source library, source table]-(task) and (task)-[relationship: target library, target table]-(), where () represents an undetermined entity. If the entity exists in the current Neo4J graph database, it is stored as the actual entity. Lineage analysis refers to the analysis of relationships between tables in big data and between other entities that rely on tables. The scheduling platform is a scheduling platform that completes tasks through a queue mechanism. A task refers to a script or code carrier that can complete a specific function. The implementation of its function may involve processing between tables, communication between systems, data output, data input, etc.

[0055] To help understand and determine the association relationship between various tasks, an example is given below.

[0056] For example, Figure 4 As shown, first, the scheduling platform log is analyzed, and the table-level lineage relationship is obtained based on the analysis results, such as (source library, source table)-[relationship: task]-(target library, target table), and this table-level lineage relationship is stored in the Neo4J graph database. At the same time, this table-level lineage relationship is disassembled to obtain task-level lineage relationship-A, that is, ()-[relationship: source library, source table]-(task) and task-level lineage relationship-B, that is, (task)-[relationship: target library, target table]-(). Finally, task-level lineage relationship-A and task-level lineage relationship-B are stored in task-level lineage relationship-B to realize the association relationship between each task and the table in the scheduling platform.

[0057] Step S20, determining the dependency relationships between the tasks based on the association relationships, and detecting the tasks based on the dependency relationships to determine whether there is an abnormal task among the tasks;

[0058] After obtaining the corresponding relationship of each task, the dependency detection will be performed on each task according to the relationship, the relationship of task dependency in the scheduling platform will be loaded, and the direct relationship of task dependency will be constructed. For example, if Task 1 depends on Task 2 and Task 3, the relationship (Task 1)-[Relationship: Export]-(Task 2) and (Task 1)-[Relationship: Export]-(Task 3) will be constructed. In this way, the dependency relationship construction of all tasks in the scheduling platform will be completed. The specific construction process is as follows: Figure 5 As shown, the scheduling platform configures tasks, traverses all task dependencies, and constructs the task relationship between each task, (Task A) - [Relationship: Export] - (Task B). This relationship is stored in the Neo4J graph database. The scheduling platform then determines whether the task has completed. If not, it continues to traverse all task dependencies. If so, the task loading is determined to be complete. Dependency detection examines the relationship between one or more entities based on tables. Dependency detection is divided into upstream dependencies, downstream dependencies, and self-dependencies.

[0059] After obtaining the dependency relationships corresponding to each task, each task can be tested based on the dependency relationships to determine whether there are any abnormal tasks among the tasks. That is, the detected dependency relationships are compared with the dependency relationships of the current task to determine whether the current task has missing dependencies, multiple dependencies, closed-loop dependencies, complex syntax, incorrect task descriptions, and other abnormal situations.

[0060] Step S30: If yes, the task with the exception is treated as an abnormal task.

[0061] Abnormal tasks refer to tasks that need to be improved in the scheduling platform, which can be divided into tasks with missing dependencies, tasks with multiple dependencies, and closed-loop tasks. If it is determined that there are tasks with abnormalities, the tasks with abnormalities will be regarded as abnormal tasks. If not, all tasks will be determined to be normal tasks. It should be noted that when detecting abnormal tasks in this embodiment, in addition to using a graph database to complete data storage and calculation of strongly connected components, it is also possible to index the dependencies in the detection task configuration table one by one, and finally complete the detection of each task. In addition, for the multiple and multiple tasks in the abnormal tasks, the association relationship between entities can be constructed through a database such as MySQL (relational database management system), and finally the two abnormal situations of multiple and multiple tasks and multiple tasks can be detected.

[0062] In this embodiment, multiple tasks in the scheduling platform are analyzed to determine the associations between each task and multiple tables in the scheduling platform; based on each association, the dependencies between the tasks are determined; and based on each dependency, each task is detected to determine whether there are any abnormal tasks among the tasks; if so, the abnormal tasks are treated as abnormal tasks. By analyzing multiple tasks in the scheduling platform, determining the dependencies between the tasks, and then detecting abnormal tasks based on the dependencies, the efficiency of detecting abnormal tasks is improved. Since it is constructed based on the relationship between tables and tasks, it also ensures that the accuracy of abnormal task detection in the scheduling platform is improved without affecting the normal operation of the tasks.

[0063] Furthermore, based on the first embodiment of the abnormal task detection method of the present invention, a second embodiment of the abnormal task detection method of the present invention is proposed. This embodiment is step S20 of the first embodiment of the present invention, and the step of detecting each of the tasks based on each of the dependencies includes:

[0064] Step a, calculating the strongly connected components corresponding to each of the tasks based on each of the dependency relationships, and detecting whether there is a closed-loop task in each of the tasks based on the strongly connected components;

[0065] This embodiment includes the analysis of closed-loop tasks in abnormal tasks, that is, after the dependency relationship between each task is established, the association information between tasks (task A)-[relationship: export]-(task B) can be used to calculate the strongly connected components between each task according to the Neo4J graph mechanism, and the results are recorded in the corresponding list. Then, it is determined whether there is a closed-loop task based on the strongly connected components, and different operations are performed according to different determination results. Among them, the algorithm for calculating the strongly connected components can adopt the Tarjan algorithm, that is, an array that records the search order; an array that records the strong connectivity; an array indicating whether a node is in the stack; and a stack that stores the search path. Among them, the strongly connected components can be to add some points and paths on the basis of the strongly connected graph, so that the current graph is no longer strongly connected, and the original strongly connected part can be considered as a strongly connected component. Strongly connected graph: If there is a loop in a directed graph (the graph of dependency relationships in the scheduling platform is a directed graph), all nodes are connected at least once, and such a graph is a strongly connected graph. For example, if Figure 6 As shown, including tasks 1-7, there are 7 nodes. These nodes and lines form a strongly connected component. It can be seen from the figure that task 5 depends on task 7, task 6 depends on task 5, and task 7 depends on task 6. At this time, it can be considered that this is a closed-loop task.

[0066] Step b: If there is a closed-loop task, the closed-loop task is regarded as an abnormal task.

[0067] When it is determined that there is a closed-loop task, the closed-loop task can be treated as an abnormal task and a corresponding prompt message can be output.

[0068] In this embodiment, by checking the tasks in the scheduling platform according to the strongly connected components to determine whether there are closed-loop tasks, the accuracy of detecting closed-loop tasks in abnormal tasks is improved.

[0069] Furthermore, the step of detecting each of the tasks based on each of the dependency relationships further includes:

[0070] Step c, obtaining the configuration dependency relationship preset in the task configuration information table corresponding to the scheduling platform;

[0071] In this embodiment, analysis of missed and over-configured tasks in abnormal tasks is also included. At this time, it is necessary to first obtain the preset configuration dependencies related to each task in the task configuration information table corresponding to the scheduling platform, and detect whether each task is a missed or over-configured task based on these configuration dependencies.

[0072] Step d, traversing each of the tasks in sequence, determining the current dependency of the currently traversed task based on each of the dependency relationships, and matching the current dependency with the configuration dependency;

[0073] Traverse each task in turn and determine the current dependency of the currently traversed task based on these dependencies. That is, through the method of (Task A)-[Relationship: Source Library, Source Table]-(Task B), it is inferred that Task B depends on Task A, thereby gradually obtaining the dependency relationships of the corresponding tasks. Then, the dependency relationships are compared with the configuration dependencies to determine whether the task is missing, which tasks are missing, and whether it is multiple configurations, and which tasks are multiple configurations.

[0074] In step e, if there is no match, the task currently being traversed is treated as a task with an exception until the traversal of each task is completed.

[0075] If it is found that the current dependency relationship and the configured dependency relationship do not match, it can be considered that the currently traversed task has a missing or over-configured phenomenon, that is, the currently traversed task is an abnormal task. It should be noted that the same detection operation needs to be performed on all tasks in the scheduling platform. For example, Figure 7As shown, first, the dependency of each task is obtained according to the task lineage relationship, such as (Task A)-[Relationship: Export]-(Task B), and stored in the Neo4J graph database. Then, the task configuration is compared, that is, the configuration dependency relationship in the task configuration information table is used to determine whether each task's task dependency has multiple or missing configurations. That is, when it is determined according to the task situation that there is a task dependency that is under-configured (i.e., missing configuration), the under-configured task is output; when it is determined according to the task situation that there is a task dependency that is over-configured, the over-configured task is output.

[0076] In this embodiment, by comparing the dependency relationship of the tasks with the configuration dependency relationship in the task configuration information table, it is determined whether there are any missed or over-configured tasks, thereby improving the accuracy of detecting abnormal tasks.

[0077] Furthermore, the step of detecting each of the tasks based on each of the dependency relationships further includes:

[0078] Step f, detecting each of the tasks in sequence based on each of the dependency relationships, parsing the target task in each of the tasks into a syntax tree using a syntax parser, and determining whether the level of the syntax tree is greater than a preset level;

[0079] In this embodiment, for a task with high maintainability, it should have high grammatical readability, and complex grammatical forms and uses should be divided into two or more different units as much as possible to implement. Among them, the complex grammar includes two levels in general. The first level is that the grammatical tree exceeds the preset level, such as 8 levels, and the second level is that the grammatical unit is complex and cannot be parsed. Of course, whether it is real-time calculation analysis or batch calculation analysis, it can be completed using a grammar parser (such as hive-exec parser). Specifically, hive-exec parser can parse an hql (Hibernate Query Language) statement into a grammar tree (the grammar tree of a general statement is within 5 levels). Of course, there are also grammar trees that cannot be parsed. For grammar trees that cannot be parsed, it means that their grammar is complex and not easy to read, and they should be listed as abnormal tasks for developers to make corrections. In the analysis process of parsing and confirming complex grammatical tasks, such as Figure 8 As shown in the figure, the target task is obtained through the text log in the relational database, and the hql of the target task is extracted through regular matching to obtain the hql statement. The hql statement is parsed into a syntax tree through a syntax parser, and it is judged whether the syntax tree parsing is successful. If not, the syntax is considered complex and the target task is an abnormal task, and it is stored in the Neo4J graph database. If so, the syntax tree depth is judged based on the table-level lineage relationship to see whether it exceeds 8 layers. If so, the syntax is considered complex and it is an abnormal task. If not, it is considered a normal task.

[0080] In step g, if it is greater than , the target task is treated as an abnormal task.

[0081] If the target task's corresponding syntax tree level is greater than the preset level, the target task can be considered an abnormal task. However, if the target task's corresponding syntax tree level is less than or equal to the preset level, the target task can be considered a normal task.

[0082] In this embodiment, by performing grammatical analysis on the tasks according to the grammatical analyzer to determine whether there are grammatically complex tasks, and treating the grammatically complex tasks as abnormal tasks, the accuracy of abnormal task detection is improved.

[0083] Furthermore, the step of detecting each of the tasks based on each of the dependency relationships further includes:

[0084] Step h, based on each of the dependency relationships, sequentially collecting the task attributes of each of the tasks, sequentially traversing each of the task attributes, performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes, and detecting whether the key information meets preset requirements;

[0085] In this embodiment, it is also possible to determine whether a task is an abnormal task by determining whether the task description conforms to the task analysis. Specifically, the task attributes (such as task name, task category, and business category) of each task are collected sequentially based on the dependencies, and the task attributes of each task are summarized to obtain summary information corresponding to each task attribute. Each summary information is then weighted and segmented to obtain key information corresponding to each task. A determination is then made as to whether the key information meets the preset requirements, and different operations are performed based on the judgment results. It should be noted that in this embodiment, the description of tasks should adhere to the following principles. The preset requirements may be: First, determine whether the task has a blood relationship through blood analysis. If there are three or more table-level blood relationships, the task should be a processing task. Second, if the task has a corresponding library, the task description should include information about the corresponding library. For example, if the corresponding library is product XXX, the description should be XXX. Third, if the task's blood dependency table has a Sang or higher association with a specific product YYY, the task description should include YYY.

[0086] In step w, if it does not meet the requirements, the task corresponding to the currently traversed task attribute is treated as an abnormal task until the traversal of the task attributes is completed.

[0087] If critical information is found to not meet the pre-set requirements, the task corresponding to the currently traversed task attributes can be considered an abnormal task, i.e., an abnormal task. However, if critical information is found to meet the pre-set requirements, the task corresponding to the currently traversed task attributes can be considered a normal task. The same operation should be performed on all tasks. It should be noted that inconsistent task descriptions will result in a task that lacks a true representation of its underlying implementation logic. Using a program to analyze the underlying logic and then reverse-check the task can help provide feedback on tasks with incomplete descriptions.

[0088] In this embodiment, the task attributes are weighted and segmented to determine whether the task description meets the task analysis. If not, it is treated as an abnormal task, thereby improving the accuracy of abnormal task detection.

[0089] Specifically, the steps of performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes include:

[0090] Step h1: Perform weighted word segmentation on the currently traversed task attributes to obtain multiple word segments corresponding to the currently traversed task attributes, and detect whether any word in each word segment is a noun;

[0091] The currently traversed task attributes are summarized to obtain summary information, and the summary information is weighted segmented to obtain multiple segmentations corresponding to the currently traversed task attributes. The weight is judged by the part of speech of each segmentation to determine whether there is a segmentation that is a noun, and different operations are performed based on different determination results.

[0092] Step h2: if it exists, increase the weight corresponding to the noun, obtain a preset number of target weights according to the weights corresponding to each of the participles, and use the participles corresponding to each of the target weights as the key information of the currently traversed task attributes.

[0093] When it is found that a segmented word is a noun, the weight corresponding to the noun can be increased according to the dictionary library. After the weight of the noun is increased, each segmented word is placed in the feature word list, and according to the size of the weight, a preset number of target weights with the largest weight are obtained, and the segmented words corresponding to these target weights are used as the key information of the task corresponding to the currently traversed task attribute. For example, Figure 9As shown, the external features of the task (i.e., task attributes, such as task name, task classification, business classification, etc.) are obtained, and then summarized to obtain summary information, and the summary information is segmented, and the weight of each segmentation is judged according to the part of speech of each segmentation to determine whether there is a segmentation for a noun. If it is a noun, the noun is queried through the dictionary library and the weight is added. If not, the weight is reduced, and all the segmentations are placed in the feature word list, and the first two segmentations with the largest weights are output.

[0094] In this embodiment, weights are added to the nouns in each participle, and then the participles corresponding to each weight are selected to determine the key information, thereby ensuring the accuracy of the acquired key information.

[0095] Furthermore, the step of analyzing the multiple tasks in the scheduling platform to determine the association relationship between each of the tasks and the multiple tables in the scheduling platform includes:

[0096] Step x, analyzing multiple tasks in the scheduling platform to obtain table-level kinship relationships between each of the tasks and multiple tables in the scheduling platform;

[0097] Analyze all tasks in the scheduling platform to obtain the table-level blood relationship between each task and the table in the scheduling platform, that is, the first-level association relationship with the task as the core, such as (source library, source table)-[relationship: task]-(target library, target table).

[0098] In step y, each of the table-level blood relationships is disassembled in a preset form to determine the association relationship between each of the tasks and the multiple tables in the scheduling platform.

[0099] After obtaining the table-level lineage relationships corresponding to each task, the table-level lineage relationships are broken down according to a preset format, and the relationships between each task and multiple tables in the scheduling platform are determined. For example, (source database, source table)-[relationship: task]-(target database, target table) can be broken down into ()-[relationship: source database, source table]->(task) and (task)-[relationship: target database, target table]->().

[0100] In this embodiment, the table-level blood relationship between the task and the table is first obtained, and then the table-level blood relationship is decomposed according to a preset form to obtain the association relationship between the task and the table, thereby ensuring the accuracy of the obtained association relationship.

[0101] The present invention also provides an abnormal task detection device, referring to Figure 3 , the abnormal task detection device includes:

[0102] A determination module, configured to analyze multiple tasks in the scheduling platform to determine associations between each of the tasks and multiple tables in the scheduling platform;

[0103] a detection module, configured to determine the dependency relationships between the tasks based on the association relationships, and detect the tasks based on the dependency relationships to determine whether there is any abnormal task among the tasks;

[0104] A module exists, for treating the task with the exception as an abnormal task if the task exists.

[0105] Furthermore, the detection module is further configured to:

[0106] Calculating the strongly connected components corresponding to each of the tasks based on each of the dependency relationships, and detecting whether there is a closed-loop task in each of the tasks based on the strongly connected components;

[0107] If a closed-loop task exists, the closed-loop task is regarded as a task with an exception.

[0108] Furthermore, the detection module is further configured to:

[0109] Obtaining a configuration dependency relationship preset in a task configuration information table corresponding to the scheduling platform;

[0110] Traversing each of the tasks in sequence, determining a current dependency of the currently traversed task based on each of the dependency relationships, and matching the current dependency with the configuration dependency;

[0111] If there is no match, the task currently being traversed is treated as a task with an exception until the traversal of each task is completed.

[0112] Furthermore, the detection module is further configured to:

[0113] Detecting each of the tasks in sequence based on each of the dependency relationships, parsing a target task in each of the tasks into a syntax tree using a syntax parser, and determining whether the level of the syntax tree is greater than a preset level;

[0114] If it is greater, the target task is treated as an abnormal task.

[0115] Furthermore, the detection module is further configured to:

[0116] Collecting the task attributes of each of the tasks in sequence based on each of the dependency relationships, traversing each of the task attributes in sequence, performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes, and detecting whether the key information meets preset requirements;

[0117] If it does not meet the requirements, the task corresponding to the currently traversed task attribute will be treated as an abnormal task until the traversal of the task attributes is completed.

[0118] Furthermore, the detection module is further configured to:

[0119] Perform weighted word segmentation on the currently traversed task attributes to obtain multiple word segmentations corresponding to the currently traversed task attributes, and detect whether any word in each of the word segmentations is a noun;

[0120] If it exists, the weight corresponding to the noun is increased, a preset number of target weights are obtained according to the size of the weights corresponding to each of the participles, and the participles corresponding to each of the target weights are used as the key information of the currently traversed task attributes.

[0121] Furthermore, the determining module is further configured to:

[0122] Analyze multiple tasks in the scheduling platform to obtain table-level kinship relationships between each task and multiple tables in the scheduling platform;

[0123] The table-level blood relationships are disassembled in a preset form to determine the association relationships between each of the tasks and the multiple tables in the scheduling platform.

[0124] The methods executed by the above program modules can refer to the various embodiments of the abnormal task detection method of the present invention, and will not be described in detail here.

[0125] The present invention also provides a computer storage medium.

[0126] The computer storage medium of the present invention stores an abnormal task detection program, which implements the steps of the abnormal task detection method described above when executed by a processor.

[0127] The method implemented when the abnormal task detection program running on the processor is executed can refer to the various embodiments of the abnormal task detection method of the present invention, and will not be described in detail here.

[0128] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0129] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0130] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0131] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting abnormal tasks, characterized in that: The abnormal task detection method comprises the following steps: Analyzing multiple tasks in the scheduling platform to determine the association relationship between each task and multiple tables in the scheduling platform; Determining dependencies between the tasks based on the association relationships, and detecting the tasks based on the dependencies to determine whether there is an abnormal task among the tasks; If so, the task with the exception is treated as an abnormal task; The detecting each of the tasks based on the dependency relationships includes: Obtain the target task through the text log in the relational database, and extract the hql of the target task through regular matching to obtain the hql statement; Use the syntax parser to parse the hql statement into a syntax tree and determine whether the syntax tree is parsed successfully; If parsing the syntax tree is unsuccessful, determining that the target task is an abnormal task; If the parsing of the syntax tree is successful, then judging whether the level of the syntax tree is greater than the preset level according to the table-level blood relationship; If it is greater, the target task is treated as an abnormal task.

2. The abnormal task detection method according to claim 1, characterized in that: The step of detecting each of the tasks based on the dependency relationships includes: Calculating the strongly connected components corresponding to each of the tasks based on each of the dependency relationships, and detecting whether there is a closed-loop task in each of the tasks based on the strongly connected components; If a closed-loop task exists, the closed-loop task is regarded as a task with an exception.

3. The abnormal task detection method according to claim 1, wherein: The step of detecting each of the tasks based on the dependency relationships includes: Obtaining a configuration dependency relationship preset in a task configuration information table corresponding to the scheduling platform; Traversing each of the tasks in sequence, determining a current dependency of the currently traversed task based on each of the dependency relationships, and matching the current dependency with the configuration dependency; If there is no match, the task currently being traversed is treated as a task with an exception until the traversal of each task is completed.

4. The abnormal task detection method according to claim 1, wherein: The step of detecting each of the tasks based on the dependency relationships includes: Collecting the task attributes of each of the tasks in sequence based on each of the dependency relationships, traversing each of the task attributes in sequence, performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes, and detecting whether the key information meets preset requirements; If it does not meet the requirements, the task corresponding to the currently traversed task attribute will be treated as an abnormal task until the traversal of the task attributes is completed.

5. The abnormal task detection method according to claim 4, characterized in that: The step of performing weighted word segmentation on the currently traversed task attributes to obtain key information of the currently traversed task attributes includes: Perform weighted word segmentation on the currently traversed task attributes to obtain multiple word segmentations corresponding to the currently traversed task attributes, and detect whether any word in each of the word segmentations is a noun; If it exists, the weight corresponding to the noun is increased, a preset number of target weights are obtained according to the size of the weights corresponding to each of the participles, and the participles corresponding to each of the target weights are used as the key information of the currently traversed task attributes.

6. The abnormal task detection method according to any one of claims 1 to 5, characterized in that: The step of analyzing the multiple tasks in the scheduling platform to determine the association relationship between each of the tasks and the multiple tables in the scheduling platform includes: Analyze multiple tasks in the scheduling platform to obtain table-level kinship relationships between each task and multiple tables in the scheduling platform; The table-level blood relationships are disassembled in a preset form to determine the association relationships between each of the tasks and the multiple tables in the scheduling platform.

7. An abnormal task detection device, characterized in that: The abnormal task detection device includes: A determination module, configured to analyze multiple tasks in the scheduling platform to determine associations between each of the tasks and multiple tables in the scheduling platform; A detection module is used to determine the dependency relationship between each of the tasks based on each of the association relationships, and to detect each of the tasks based on each of the dependency relationships to determine whether there is an abnormal task among the tasks; the step of detecting each of the tasks based on each of the dependency relationships includes: obtaining a target task through a text log in a relational database, and extracting the hql of the target task through regular matching to obtain an hql statement; parsing the hql statement into a syntax tree through a syntax parser, and judging whether the parsing of the syntax tree is successful; if the parsing of the syntax tree is unsuccessful, determining that the target task is an abnormal task; if the parsing of the syntax tree is successful, judging whether the level of the syntax tree is greater than a preset level based on the table-level blood relationship; if it is greater, treating the target task as an abnormal task; A module exists, for treating the task with the exception as an abnormal task if the task exists.

8. An abnormal task detection device, characterized in that: The abnormal task detection device includes: a memory, a processor, and an abnormal task detection program stored in the memory and executable on the processor. When the abnormal task detection program is executed by the processor, the steps of the abnormal task detection method according to any one of claims 1 to 6 are implemented.

9. A computer storage medium, characterized in that The computer storage medium stores an abnormal task detection program, which, when executed by a processor, implements the steps of the abnormal task detection method according to any one of claims 1 to 6.

Images