Task dependency correction method, device, computer equipment and storage medium

Abstract

The embodiment of the present application belongs to the field of R&D management, and relates to a task dependency correction method, device, computer equipment and storage medium, the method comprising: obtaining the initial task dependency information of each task; extracting the task identifier from the initial task dependency information; obtaining the task script corresponding to the task identifier, and obtaining the underlying file access log based on the task identifier; correcting the initial task dependency information according to the task script and the underlying file access log to obtain the task dependency information. In addition, the present application also relates to blockchain technology, and the initial task dependency information and the task dependency information can be stored in the blockchain. The present application improves the accuracy of task dependency correction.

Description

Technical Field

[0001] The present application relates to the field of R&D management technology, and in particular to a task dependency correction method, device, computer equipment and storage medium. Background Art

[0002] Production and operation activities are often divided into tasks. For example, in a big data platform, there may be tens of thousands of tasks at the same time. Due to the data dependency between different tasks, there is a dependency between tasks. For example, Task A processes the source data to obtain Data Table A, which will be used by Task B. Task B can only run after Task A has completed running. At this time, Task B depends on Task A.

[0003] In addition to direct dependencies, there may also be indirect dependencies between tasks. Accurately configuring task dependencies and correcting incorrect task dependencies are of great significance to the accurate and efficient operation of tasks. However, the current verification of task dependencies is usually based on manual implementation. As the number of tasks increases and task dependencies become more complex, the accuracy of manual correction of task dependencies is getting lower and lower. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a task dependency correction method, apparatus, computer equipment and storage medium to solve the problem of low accuracy of task dependency.

[0005] In order to solve the above technical problems, the present application embodiment provides a task dependency correction method, which adopts the following technical solution:

[0006] Get the initial task dependency information of each task;

[0007] Extracting a task identifier from the initial task dependency information;

[0008] Obtaining the task script corresponding to the task identifier, and obtaining the underlying file access log based on the task identifier;

[0009] The initial task dependency information is corrected according to the task script and the underlying file access log to obtain task dependency information.

[0010] In order to solve the above technical problems, the embodiment of the present application further provides a task dependency correction device, which adopts the following technical solution:

[0011] An information acquisition module, used to obtain the initial task dependency information of each task;

[0012] An identification extraction module, used to extract a task identification from the initial task dependency information;

[0013] An acquisition module, used to acquire the task script corresponding to the task identifier, and acquire the underlying file access log based on the task identifier;

[0014] The dependency correction module is used to correct the initial task dependency information according to the task script and the underlying file access log to obtain task dependency information.

[0015] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0016] Get the initial task dependency information of each task;

[0017] Extracting a task identifier from the initial task dependency information;

[0018] Obtaining the task script corresponding to the task identifier, and obtaining the underlying file access log based on the task identifier;

[0019] The initial task dependency information is corrected according to the task script and the underlying file access log to obtain task dependency information.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0021] Get the initial task dependency information of each task;

[0022] Extracting a task identifier from the initial task dependency information;

[0023] Obtaining the task script corresponding to the task identifier, and obtaining the underlying file access log based on the task identifier;

[0024] The initial task dependency information is corrected according to the task script and the underlying file access log to obtain task dependency information.

[0025] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: obtaining initial task dependency information that records the dependencies between tasks, and extracting the task identifier therefrom; obtaining the task script and the underlying file access log according to the task identifier, wherein the task script records the actions performed by the task, thereby obtaining the reading and processing of the data table when the task is executed, and the underlying file access log records the access information to the underlying data table of the system when the task is executed. The tasks generate dependencies through the data table, so the dependencies between tasks can be judged through the task script and the underlying file access log, thereby automatically correcting the initial task dependency information, thereby improving the accuracy of the generated task dependency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0028] Figure 2 is a flow chart of an embodiment of a task dependency correction method according to the present application;

[0029] Figure 3 is a structural schematic diagram of an embodiment of a task dependency correction device according to the present application;

[0030] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0037] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0038] It should be noted that the task dependency correction method provided in the embodiment of the present application is generally executed by a server, and accordingly, the task dependency correction device is generally arranged in the server.

[0039] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Continue to refer Figure 2 , shows a flow chart of an embodiment of a task dependency correction method according to the present application. The task dependency correction method comprises the following steps:

[0041] Step S201, obtaining initial task dependency information of each task.

[0042] In this embodiment, the task dependency correction method is executed on the electronic device (eg Figure 1 The server shown in the figure) can communicate with the terminal through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0043] Specifically, the server first obtains the initial task dependency information of each task, wherein the initial task dependency information records the dependency between each task. In one embodiment, the task is a task on a big data platform, such as an ETL (Extract-Transform-Load, which refers to the process of extracting, transforming, and loading data from the source to the destination) task. When the ETL task is launched on the big data platform, the dependency between the ETL tasks can be manually added to obtain the initial task dependency information.

[0044] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned initial task dependency information, the above-mentioned initial task dependency information can also be stored in a node of a blockchain; it can be understood that in order to further ensure the privacy and security of the task dependency information finally generated by this application, the above-mentioned task dependency information can also be stored in a node of a blockchain.

[0045] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0046] Step S202: extracting a task identifier from the initial task dependency information.

[0047] Specifically, the initial task dependency information may be structured data, from which a task identifier may be extracted. The task identifier is an identifier of each task, is unique, and can be used to distinguish each task.

[0048] Step S203, obtaining the task script corresponding to the task identifier, and obtaining the underlying file access log based on the task identifier.

[0049] Specifically, the task has a task script, and the task is issued and executed by executing the task script. The task script has a task identifier, and after the task identifier is extracted from the initial task dependency information, the corresponding task script is searched for in multiple pre-stored task scripts according to the task identifier.

[0050] At the same time, the underlying file access log is obtained according to the task identifier. A task log is generated when the task is running. The underlying file access log can be a part of the task log, which records the operations on the underlying data table when the task is running.

[0051] Step S204: Correct the initial task dependency information according to the task script and the underlying file access log to obtain the task dependency information.

[0052] Specifically, the task script records the actions performed when the task is running in a scripting language, including operations such as reading, writing, and generating data tables; the underlying file access log records the operations on the underlying data tables when the task is running in the form of a log.

[0053] When a task is running, it is necessary to operate on the data table. The data in one data table can come from another data table. Therefore, the dependency between tasks can be obtained through the relationship between tasks and data tables recorded in the task script and the underlying file access log, so as to automatically verify and correct the initial task dependency information and obtain the task dependency information.

[0054] In this embodiment, initial task dependency information that records the dependencies between tasks is obtained, and the task identifier is extracted therefrom; the task script and the underlying file access log are obtained based on the task identifier, wherein the task script records the actions performed by the task, thereby obtaining the reading and processing of the data table when the task is executed, and the underlying file access log records the access information to the underlying data table of the system when the task is executed. The tasks generate dependencies through the data table, so the dependencies between tasks can be judged through the task script and the underlying file access log, thereby automatically correcting the initial task dependency information, thereby improving the accuracy of the generated task dependency information.

[0055] Furthermore, the step of obtaining the underlying file access log based on the task identifier may include: obtaining time limit information; and searching the underlying file access log in the log library according to the time limit information and the task identifier.

[0056] Specifically, the underlying file access log is a log, and the task can be executed repeatedly, and each execution will generate a task log. Therefore, the task log has time limit information, which is used to limit and explain the time dimension of the log. For example, when running a task on May 2, 2022, the data processed is May 1, 2022, and the incremental date "May 1, 2022" is used as the time limit information.

[0057] Therefore, time limit information can also be obtained, and based on the time limit information and the task identifier, the required underlying file access log can be obtained from a large number of logs stored in the log library.

[0058] In one embodiment, based on the task status information in the task log, the log generated by the normally running task is selected, the running time of the task is used as the time limit information, and then the underlying file access log is queried based on the time limit information and the task identifier, thereby ensuring the accuracy of the underlying file access log.

[0059] In this embodiment, the required underlying file access log can be accurately obtained from the log library according to the time limit information and the task identifier.

[0060] Furthermore, before the above step S204, the following may also be included: when each task is running, each task is monitored to record the underlying data table accessed when each task is running, and obtain an underlying file access log.

[0061] Specifically, the big data platform of the present application has been improved so that when each task is running, the operation of each task can be monitored throughout the entire process, and the operations on the underlying data table are fully recorded to obtain the underlying file access log. For example, when a task operates a Hadoop task in a hive (hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. Hadoop is a big data software system operating framework.) database, the background together with the task identification, time limit information (such as date) and the operation of the HDFS (a distributed file system in Hadoop) file are recorded to obtain the HDFS access information of the task to the underlying data table. If it is a partitioned table, it needs to be accurate to the partition path. The underlying file access log can more accurately reflect the execution of the task.

[0062] In this embodiment, each task is monitored to record the underlying data table accessed when the task is running, and the underlying file access log is obtained, which can more accurately reflect the operation status of the task on the underlying data table.

[0063] Furthermore, the above-mentioned step S204 may include: identifying each read-write statement in the task script; parsing each read-write statement to obtain a read-write parsing result; when it is determined according to the read-write parsing result that each read-write statement is a first type of statement, correcting the initial task dependency information through the read-write parsing result to obtain task dependency information; when it is determined according to the read-write parsing result that there is a second type of statement in each read-write statement, correcting the initial task dependency information through the read-write parsing result and the underlying file access log to obtain task dependency information.

[0064] Specifically, the task script can be processed first to identify each read and write statement in the task script. The statements in the task script can be HQL statements, and the script statements can show what the script does at each step, including which data table to obtain data from, which data table to modify, and which data table to generate. Parsing each read and write statement can obtain the above information and generate a read and write parsing result.

[0065] During the parsing process, the attribute information of the data table can also be obtained, and the read statement can be divided into the first type of statement and the second type of statement according to the attribute information of the data table. If the data table operated by the read statement is a conventional type of data table, the read statement is a first type of statement, and if the data table operated by the read statement is an unconventional type of data table, the read statement is a second type of statement. Among them, the conventional type of data table means that the data table involved in the statement literally is consistent with the underlying data table actually stored in the database, and the unconventional type of data table means that the data table involved in the statement literally is inconsistent with the underlying data table actually stored in the database.

[0066] In a read operation, a view may be involved. A view is a view in a computer database, which is a virtual table whose content is defined by a query. Like a real table, a view contains a series of named columns and row data. However, a view does not exist in the database as a stored set of data values. The row and column data come from the table referenced by the query that defines the view, and are dynamically generated when the view is referenced. A view is a typical unconventional type of data table. Therefore, if the data table parsed to the read statement operation does not include a view, the read statement can be marked as a first type statement, and if the data table parsed to the read statement operation includes a view, the read statement is marked as a second type statement.

[0067] Therefore, when all read and write statements are first-type statements, the initial task dependency can be corrected only based on the read and write analysis results to obtain task dependency information; if there are second-type statements in each read and write statement, it is also necessary to correct them based on the underlying file access log to obtain task dependency information.

[0068] In this embodiment, the read-write statements in the task script are identified and parsed, and the statement types of the read-write statements are determined according to the read-write parsing results, so that corresponding correction strategies are adopted to ensure the accuracy of task dependency correction.

[0069] Furthermore, the above-mentioned step of parsing each read-write statement to obtain a read-write parsing result may include: parsing each read-write statement to obtain an initial read-write parsing result; when it is determined according to the initial read-write parsing result that there are multiple dependent data tables, extracting task description information from the task script; the multiple dependent data tables are dependent on multiple tasks; adding time sequence information to the initial read-write parsing result according to the task description information to obtain a read-write parsing result.

[0070] Specifically, each read-write statement is parsed to obtain an initial read-write parsing result. The initial read-write parsing result can show which tasks need to read and write the data table. If a data table is read and written by multiple tasks, the data table is dependent on multiple tasks and is a multi-dependent data table.

[0071] For multi-dependent data tables, if multiple tasks read and write to them at the same time, data errors may occur. At this time, it is also necessary to extract task description information from the task script. The task description information is used to record the timeliness information and criticality of the task. The timeliness information records the urgency of the task or the order of execution, and the criticality reflects the importance of the task.

[0072] According to the task description information, the time sequence of the task operation data table can be determined, and the execution of the tasks can be staggered to obtain the time sequence information; the time sequence information is added to the initial read-write parsing result to obtain the read-write parsing result, thereby ensuring the accuracy of the read-write parsing result.

[0073] In this embodiment, each read-write statement is parsed to obtain an initial read-write parsing result. When there are multiple dependent data tables, description information is obtained, and time sequence information is added to the initial statement parsing result according to the task description information, thereby staggering the execution order of the tasks and improving the accuracy of the read-write parsing results.

[0074] Furthermore, the above-mentioned step of correcting the initial task dependency information through the read-write parsing results and the underlying file access log to obtain the task dependency information may include: based on the read-write parsing results and the underlying file access log, obtaining first dependency information recording the dependency between the underlying data tables, and obtaining second dependency information recording the dependency between the task and the underlying data table; correcting the initial task dependency information according to the first dependency information and the second dependency information to obtain the task dependency information.

[0075] Specifically, based on the read and write analysis results and the underlying file access logs, the read and write operations of the task on the underlying data table can be clearly and accurately obtained. Therefore, there is a dependency relationship between the task and the underlying data table; at the same time, the data in the underlying data table can come from another underlying data table, so there is also a dependency relationship between the underlying data tables.

[0076] The dependency between the underlying data tables is recorded as the first dependency information, and the dependency between the task and the underlying data table is recorded as the second dependency information. It can be understood that the dependency between tasks originates from the underlying data table. Therefore, the dependency between tasks can be sorted out according to the first dependency information and the second dependency information, so as to correct the initial task dependency information and obtain the task dependency information.

[0077] It can be understood that when each read-write statement is a first-type statement, the initial task dependency information is corrected through the read-write parsing results. The logic is the same as that through the read-write parsing results and the underlying file access log. The only difference is that when there are second-type statements in each read-write statement, in order to prevent interference caused by views, etc., it is necessary to add the underlying file access log for judgment.

[0078] Correction of the initial task dependency information includes removing redundant dependencies and adding missing dependencies. For example, Task A is configured with two dependencies: Task B and Task C, and Task B is configured with one dependency: Task C. Through the read-write parsing results and the underlying file access log, it is determined that Task B has indeed accessed the data table generated by Task C, and the initial task dependency information has recorded that Task B depends on Task C, and Task A depends on Task B. Then the actual execution order of the tasks from front to back should be Task C, Task B, and Task A. At this time, there is no need to configure Task A to depend on Task C, which is a redundant dependency. It is only necessary to configure Task A to depend on Task B, because Task A actually depends on Task C indirectly. At this time, "Task A depends on Task C" in the initial task dependency information can be deleted. For another example, Task D actually only uses the data table generated by Task B, but the initial task dependency information also configures Task D to depend on Task A. Then Task D needs to wait meaninglessly when executing, and will not start until Task A is completed, which affects the execution time of Task E. At this time, "Task D depends on Task A" in the initial task dependency information can be deleted.

[0079] The situation of missing dependencies is: the initial task dependency information records that task A depends on task B, and task B depends on task C. Through the read-write parsing results and the underlying file access log, it is found that task A accesses the underlying data table D, and the underlying data table D is processed and generated by task D. However, the initial task dependency information does not configure task A / B / C to depend on task D. Therefore, it can be determined that the missing task A depends on task D, and it is added to the initial task dependency information to complete the dependency.

[0080] In this embodiment, based on the read-write parsing results and the underlying file access log, first dependency information recording the dependency between the underlying data tables and second dependency information recording the dependency between the tasks and the underlying data tables are obtained; the dependencies between the tasks are inferred based on the first dependency information and the second dependency information, so as to discover redundant or missing dependencies between the tasks and correct them, thereby improving the accuracy of the task dependency information.

[0081] Furthermore, after the above step S204, the following may also be included: obtaining the current task to be executed; determining the dependent tasks of the task to be executed according to the task dependency information; performing completion check on the dependent tasks; and issuing the task to be executed when the dependent tasks pass the completion check.

[0082] Specifically, the current tasks to be executed in the big data platform are obtained, and the corrected task dependency information is obtained, and the dependent tasks of the tasks to be executed, that is, the tasks on which the tasks to be executed depend, are queried through the task dependency information.

[0083] The completion check is performed on the queried dependent tasks. The completion check is to check whether the task has been completed and whether it has been successfully executed. When the task is completed and successfully executed, the data tables operated by the dependent tasks are in a normal state, and no errors will occur when the tasks to be executed operate on these data tables again. Therefore, when it is determined that the dependent tasks have passed the completion check, it is determined that the tasks to be executed can be executed, and the tasks to be executed are sent for execution.

[0084] In this embodiment, dependent tasks of the task to be executed are obtained based on the task dependency information and completion verification is performed. When the dependent task passes the completion verification, it indicates that the data table on which the task to be executed depends is correct. At this time, the task to be executed is issued to ensure the accuracy of the execution of the task to be executed.

[0085] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology, such as identifying read and write statements in task scripts, and parsing read and write statements. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0086] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0087] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0088] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0089] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a task dependency correction device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0090] like Figure 3As shown, the task dependency correction device 300 described in this embodiment includes: an information acquisition module 301, an identification extraction module 302, an acquisition module 303 and a dependency correction module 304, wherein:

[0091] The information acquisition module 301 is used to acquire the initial task dependency information of each task.

[0092] The identifier extraction module 302 is used to extract the task identifier from the initial task dependency information.

[0093] The acquisition module 303 is used to acquire the task script corresponding to the task identifier, and acquire the underlying file access log based on the task identifier.

[0094] The dependency correction module 304 is used to correct the initial task dependency information according to the task script and the underlying file access log to obtain the task dependency information.

[0095] In this embodiment, initial task dependency information that records the dependencies between tasks is obtained, and the task identifier is extracted therefrom; the task script and the underlying file access log are obtained based on the task identifier, wherein the task script records the actions performed by the task, thereby obtaining the reading and processing of the data table when the task is executed, and the underlying file access log records the access information to the underlying data table of the system when the task is executed. The tasks generate dependencies through the data table, so the dependencies between tasks can be judged through the task script and the underlying file access log, thereby automatically correcting the initial task dependency information, thereby improving the accuracy of the generated task dependency information.

[0096] In some optional implementations of this embodiment, the acquisition module 303 may include: a restricted acquisition submodule and a log query submodule, wherein:

[0097] The restriction acquisition submodule is used to obtain time limit information.

[0098] The log query submodule is used to query the underlying file access log in the log library according to the time limit information and task identifier.

[0099] In this embodiment, the required underlying file access log can be accurately obtained from the log library according to the time limit information and the task identifier.

[0100] In some optional implementations of this embodiment, the task dependency correction device 300 may also include: an operation recording module, which is used to monitor each task when it is running to record the underlying data table accessed when each task is running, and obtain an underlying file access log.

[0101] In this embodiment, each task is monitored to record the underlying data table accessed when the task is running, and the underlying file access log is obtained, which can more accurately reflect the operation status of the task on the underlying data table.

[0102] In some optional implementations of this embodiment, the dependency correction module 304 may include: a sentence recognition submodule, a sentence parsing submodule, a first correction submodule, and a second correction submodule, wherein:

[0103] The statement recognition submodule is used to recognize each read and write statement in the task script.

[0104] The statement parsing submodule is used to parse each read and write statement to obtain the read and write parsing results.

[0105] The first correction submodule is used to correct the initial task dependency information according to the read-write parsing result to obtain the task dependency information when it is determined that all the read-write statements are first-type statements according to the read-write parsing result.

[0106] The second correction submodule is used to correct the initial task dependency information through the read-write parsing result and the underlying file access log to obtain the task dependency information when it is determined that the second type of statement exists in each read-write statement according to the read-write parsing result.

[0107] In this embodiment, the read-write statements in the task script are identified and parsed, and the statement types of the read-write statements are determined according to the read-write parsing results, so that corresponding correction strategies are adopted to ensure the accuracy of task dependency correction.

[0108] In some optional implementations of this embodiment, the sentence parsing submodule may include: a sentence parsing unit, a description extraction unit, and a time adding unit, wherein:

[0109] The statement parsing unit is used to parse each read-write statement to obtain an initial read-write parsing result.

[0110] The description extraction unit is used to extract task description information from the task script when it is determined according to the initial read-write parsing result that there are multiple dependent data tables; the multiple dependent data tables are dependent on multiple tasks.

[0111] The time adding unit is used to add time sequence information to the initial read-write parsing result according to the task description information to obtain the read-write parsing result.

[0112] In this embodiment, each read-write statement is parsed to obtain an initial read-write parsing result. When there are multiple dependent data tables, description information is obtained, and time sequence information is added to the initial statement parsing result according to the task description information, thereby staggering the execution order of the tasks and improving the accuracy of the read-write parsing results.

[0113] In some optional implementations of this embodiment, the second syndrome submodule may include: a dependency acquisition unit and a dependency correction unit, wherein:

[0114] The dependency acquisition unit is used to acquire first dependency information recording the dependency between underlying data tables based on the read-write parsing results and the underlying file access log, and acquire second dependency information recording the dependency between the task and the underlying data table.

[0115] The dependency correction unit is used to correct the initial task dependency information according to the first dependency information and the second dependency information to obtain the task dependency information.

[0116] In this embodiment, based on the read-write parsing results and the underlying file access log, first dependency information recording the dependency between the underlying data tables and second dependency information recording the dependency between the tasks and the underlying data tables are obtained; the dependencies between the tasks are inferred based on the first dependency information and the second dependency information, so as to discover redundant or missing dependencies between the tasks and correct them, thereby improving the accuracy of the task dependency information.

[0117] In some optional implementations of this embodiment, the task dependency correction device 300 may further include: a task acquisition module, a dependency determination module, a completion verification module, and a task delivery module, wherein:

[0118] The task acquisition module is used to obtain the current tasks to be executed.

[0119] The dependency determination module is used to determine the dependent tasks of the task to be executed according to the task dependency information.

[0120] The completion verification module is used to verify the completion of dependent tasks.

[0121] The task delivery module is used to deliver tasks to be executed when dependent tasks pass the completion check.

[0122] In this embodiment, dependent tasks of the task to be executed are obtained based on the task dependency information and completion verification is performed. When the dependent task passes the completion verification, it indicates that the data table on which the task to be executed depends is correct. At this time, the task to be executed is issued to ensure the accuracy of the execution of the task to be executed.

[0123] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0124] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0125] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0126] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the task dependency correction method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0127] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the task dependency correction method.

[0128] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0129] The computer device provided in this embodiment can execute the above-mentioned task dependency correction method. Here, the task dependency correction method can be the task dependency correction method of each of the above-mentioned embodiments.

[0130] In this embodiment, initial task dependency information that records the dependencies between tasks is obtained, and the task identifier is extracted therefrom; the task script and the underlying file access log are obtained based on the task identifier, wherein the task script records the actions performed by the task, thereby obtaining the reading and processing of the data table when the task is executed, and the underlying file access log records the access information to the underlying data table of the system when the task is executed. The tasks generate dependencies through the data table, so the dependencies between tasks can be judged through the task script and the underlying file access log, thereby automatically correcting the initial task dependency information, thereby improving the accuracy of the generated task dependency information.

[0131] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the task dependency correction method as described above.

[0132] In this embodiment, initial task dependency information that records the dependencies between tasks is obtained, and the task identifier is extracted therefrom; the task script and the underlying file access log are obtained based on the task identifier, wherein the task script records the actions performed by the task, thereby obtaining the reading and processing of the data table when the task is executed, and the underlying file access log records the access information to the underlying data table of the system when the task is executed. The tasks generate dependencies through the data table, so the dependencies between tasks can be judged through the task script and the underlying file access log, thereby automatically correcting the initial task dependency information, thereby improving the accuracy of the generated task dependency information.

[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0134] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A task dependency correction method, characterized in that: The steps include: Get the initial task dependency information of each task; Extracting a task identifier from the initial task dependency information; Obtaining the task script corresponding to the task identifier, and obtaining the underlying file access log based on the task identifier; wherein the underlying file access log is a part of the task log, recording the operation on the underlying data table when the task is running; Correcting the initial task dependency information according to the task script and the underlying file access log to obtain task dependency information; The step of correcting the initial task dependency information according to the task script and the underlying file access log to obtain the task dependency information includes: Identify each read and write statement in the task script; Parsing each of the read and write statements to obtain a read and write parsing result; When it is determined according to the read-write parsing result that all the read-write statements are first-type statements, the initial task dependency information is corrected according to the read-write parsing result to obtain task dependency information; When it is determined according to the read-write parsing result that there are second-type statements in each of the read-write statements, the initial task dependency information is corrected by using the read-write parsing result and the underlying file access log to obtain task dependency information; The step of parsing the read and write statements to obtain the read and write parsing results includes: Parsing each of the read and write statements to obtain an initial read and write parsing result; When it is determined according to the initial read-write parsing result that there is a multi-dependent data table, extracting task description information from the task script; the multi-dependent data table is dependent on multiple tasks; Add time sequence information to the initial read-write parsing result according to the task description information to obtain a read-write parsing result.

2. The task dependency correction method according to claim 1, characterized in that: The step of obtaining the underlying file access log based on the task identifier includes: Get time limit information; According to the time limit information and the task identifier, the underlying file access log is queried in the log library.

3. The task dependency correction method according to claim 1, characterized in that: Before the step of correcting the initial task dependency information according to the task script and the underlying file access log to obtain the task dependency information, the step further includes: When each task is running, each task is monitored to record the underlying data table accessed during the running of each task, and obtain an underlying file access log.

4. The task dependency correction method according to claim 1, characterized in that: The step of correcting the initial task dependency information by using the read-write parsing result and the underlying file access log to obtain the task dependency information comprises: Based on the read-write parsing result and the underlying file access log, first dependency information recording the dependency between underlying data tables is obtained, and second dependency information recording the dependency between tasks and underlying data tables is obtained; The initial task dependency information is corrected according to the first dependency information and the second dependency information to obtain task dependency information.

5. The task dependency correction method according to claim 1, characterized in that: After the step of correcting the initial task dependency information according to the task script and the underlying file access log to obtain the task dependency information, the method further includes: Get the current tasks to be executed; Determine the dependent tasks of the task to be executed according to the task dependency information; Performing completion check on the dependent tasks; When the dependent task passes the completion check, the task to be executed is issued.

6. A task dependency correction device, characterized in that: include: An information acquisition module, used to obtain the initial task dependency information of each task; An identification extraction module, used to extract a task identification from the initial task dependency information; An acquisition module, used to acquire the task script corresponding to the task identifier, and acquire the underlying file access log based on the task identifier; wherein the underlying file access log is a part of the task log, recording the operation on the underlying data table when the task is running; A dependency correction module, used to correct the initial task dependency information according to the task script and the underlying file access log to obtain task dependency information; The dependency correction module includes: a sentence recognition submodule, a sentence parsing submodule, a first correction submodule and a second correction submodule, wherein: The statement recognition submodule is used to recognize each read and write statement in the task script; The statement parsing submodule is used to parse each read and write statement and obtain the read and write parsing results; A first correction submodule, configured to correct the initial task dependency information by using the read-write parsing result to obtain the task dependency information when it is determined that all the read-write statements are first-type statements according to the read-write parsing result; A second correction submodule is used for correcting the initial task dependency information through the read-write parsing result and the underlying file access log to obtain the task dependency information when it is determined that the second type of statement exists in each read-write statement according to the read-write parsing result; The sentence parsing submodule includes: a sentence parsing unit, a description extraction unit and a time adding unit, wherein: A statement parsing unit, used to parse each read-write statement and obtain an initial read-write parsing result; A description extraction unit, used to extract task description information from the task script when it is determined according to the initial read-write parsing result that there are multiple dependent data tables; the multiple dependent data tables are dependent on multiple tasks; The time adding unit is used to add time sequence information to the initial read-write parsing result according to the task description information to obtain the read-write parsing result.

7. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the task dependency correction method according to any one of claims 1 to 5 when executing the computer-readable instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the task dependency correction method according to any one of claims 1 to 5 are implemented.

Images