Operation processing broken chain identification method and device

Abstract

The invention discloses a job processing broken chain identification method and device, which can be used in the technical field of information security, and the method comprises the following steps: obtaining fields extracted from production state job logs in different production environment databases, and storing the fields in the databases; analyzing SQL statements of production state job logs in the database, and constructing data consanguinity for each SQL statement; performing semantic analysis and correction on the data blood relationship in combination with the ontology knowledge base; according to all the data consanguinity, forming a data processing link, and obtaining an SQL execution sequence corresponding to the data processing link; fields of production state job logs stored in a database are matched with the SQL execution sequence, and a production state job execution sequence is obtained; and checking the production state operation execution sequence according to the operation configuration information, and determining the chain breakage point of the configuration state operation processing chain. According to the method, the breakpoint identification of the configuration state operation chain can be accurately and automatically configured in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and particularly to a method and device for identifying broken links in job processing. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention stated in the claims. The descriptions herein are not admitted to be prior art merely because they are included in this section.

[0003] With the development of technologies such as the Internet, big data, and artificial intelligence, the digital transformation of banking services has become an inevitable trend. In this process, data is the core of banking services, and the data processing link has become the key to bank operations. Banks rely on the processing and analysis of a large amount of data to drive decision-making and customer service, which involves important data such as customer information, transaction records, account balances, etc. For a bank, with the rapid growth of business, the number of data processing jobs can reach more than 3 million. Banking services generally take data as the unit, and the job processing link (here referring to the configured job processing chain) is complex, with a large amount of data and numerous processing links. The configured job processing chain may lead to data loss, processing delays or errors, thereby affecting the accuracy and efficiency of banking services. The existing job processing systems in banks often lack an effective link monitoring mechanism, making it difficult to quickly discover and locate problems, and affecting the service recovery time. When problems occur in job processing, operation and maintenance personnel need to manually check complex job configurations, a large number of job execution production job logs and process records. This approach is inefficient, showing problems such as insufficient real-time performance, low accuracy, and low automation, making it difficult to achieve real-time monitoring and rapid response in the event of job processing link failures, and unable to meet the strict real-time requirements of banking services. The job processing link in banks is complex, involving multiple systems, multiple links, and multiple data types. Existing methods are difficult to comprehensively and accurately manage and analyze these complex links, resulting in incomplete or inaccurate broken link identification. Summary of the Invention

[0004] Embodiments of the present invention provide a method for identifying broken links in job processing, which is used to achieve real-time, accurate, and automated breakpoint identification of configured job chains based on production data lineage and production job logs, ensure the stability and security of the configured job processing chain, reduce maintenance costs, reduce human factors, and improve the operation efficiency of banking services and the quality of customer service. The method includes:

[0005] Obtain production job logs in different production environment databases, and store the fields extracted from all production job logs in a database;

[0006] Parse the SQL statements of the production job logs in the database, and construct data lineage for each SQL statement;

[0007] Combine with the ontology knowledge base to perform semantic analysis and correction on data lineage, where the ontology knowledge base is used to store the definitions and relationships of various concepts related to jobs;

[0008] Form a data processing link according to all data lineage and obtain the SQL execution order corresponding to the data processing link;

[0009] Match the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order;

[0010] Check the production state job execution order according to the job configuration information to determine the break points of the configured state job processing chain, where the configured state job processing chain is determined according to the configured state job execution order in the job configuration information.

[0011] An embodiment of the present invention further provides a job processing break point identification device, which is used to realize real-time, accurate, and automated identification of break points of the configured state job chain based on production state data lineage and production state job logs, ensure the stability and security of the configured state job processing chain, reduce maintenance costs, reduce human factors, and improve the operation efficiency of banking services and customer service quality. The device includes:

[0012] A production state job log acquisition module, which is used to acquire production state job logs in different production environment databases and store the fields extracted from all production state job logs in the database;

[0013] A data lineage analysis module, which is used to analyze the SQL statements of the production state job logs in the database and construct data lineage for each SQL statement;

[0014] A data lineage correction module, which is used to combine with the ontology knowledge base to perform semantic analysis and correction on data lineage, where the ontology knowledge base is used to store the definitions and relationships of various concepts related to jobs;

[0015] A data processing link formation module, which is used to form a data processing link according to all data lineage and obtain the SQL execution order corresponding to the data processing link;

[0016] A production state job execution order acquisition module, which is used to match the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order;

[0017] A break point identification module, which is used to check the production state job execution order according to the job configuration information to determine the break points of the configured state job processing chain, where the configured state job processing chain is determined according to the configured state job execution order in the job configuration information.

[0018] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-mentioned job processing disconnection recognition method is implemented.

[0019] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned job processing disconnection recognition method is implemented.

[0020] An embodiment of the present invention further provides a computer program product including a computer program, and when the computer program is executed by a processor, the above-mentioned job processing disconnection recognition method is implemented.

[0021] In an embodiment of the present invention, production state job logs in different production environment databases are obtained, and the fields extracted from all production state job logs are stored in a database; the SQL statements of the production state job logs in the database are parsed, and data lineage is constructed for each SQL statement; in combination with an ontology knowledge base for storing definitions and relationships of various concepts related to jobs, semantic analysis and correction are performed on the data lineage; according to all the data lineage, a data processing link is formed, and the SQL execution order corresponding to the data processing link is obtained; the fields of the production state job logs stored in the database are matched with the SQL execution order to obtain the production state job execution order; according to the job configuration information, the production state job execution order is checked to determine the disconnection points of the configured state job processing chain, where the configured state job processing chain is determined according to the configured state job execution order in the job configuration information. Compared with the conventional disconnection recognition that requires operation and maintenance personnel to manually check the job configuration, a large number of job execution logs, process records, and business processing rules, resulting in low efficiency, showing problems of insufficient real-time performance, low accuracy, and low automation. The embodiment of the present invention uses production state logs to construct the SQL execution order corresponding to the data lineage, that is, the data processing relationship actually and completely processed in production, which has real-time performance, effectiveness, and accuracy, and eliminates the invalid link processing relationships. Matching the SQL execution order with the fields of the production state job logs to obtain the production state job execution order can cover various database products, and only need to ensure that the database logs can be normally output, and can accurately, quickly, and automatically give the job processing link disconnection list. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0023] Figure 1 It is a flowchart of the operation processing chain break identification method in the embodiment of the present invention;

[0024] Figure 2 It is a flowchart of constructing data lineage for each SQL statement in the embodiment of the present invention;

[0025] Figure 3 It is a schematic structural diagram of the abstract syntax tree in the embodiment of the present invention;

[0026] Figure 4 It is a flowchart of semantic analysis and correction of data lineage in the embodiment of the present invention;

[0027] Figure 5 It is a flowchart of forming a data processing link in the embodiment of the present invention;

[0028] Figure 6 It is the correspondence between the SQL execution order and the data processing link in the embodiment of the present invention;

[0029] Figure 7 It is a flowchart of determining the chain break point of the configured state operation processing chain in the embodiment of the present invention;

[0030] Figure 8 It is a schematic structural diagram of the operation processing chain break identification device in the embodiment of the present invention;

[0031] Figure 9 It is a schematic diagram of the computer device in the embodiment of the present invention. Detailed implementation manners

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the following will further elaborate on the embodiments of the present invention in conjunction with the drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0033] In the technical solutions of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations.

[0034] It should be noted that in the embodiments of the present application, some existing industry solutions such as certain software, components, models, etc. may be mentioned. They should be regarded as exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solutions of the present application, but it does not mean that the applicant has already or necessarily used this solution.

[0035] First, the terms involved in the embodiments of the present invention are explained.

[0036] Job: A job is the basic configuration unit of a job control and scheduling system, that is, a logical unit defined by the user to complete a certain task, and it includes the executed program and its parameters. In task scheduling, a job is the smallest execution unit. Different parameter forms of the same executed program are different job configurations for the system.

[0037] Job dependency: A job has a previous dependency on one or more jobs / conditions. Only after the current job is executed or the conditions are met can the next job run.

[0038] SQL: It is a structured query language that runs on a database and returns the results queried from the database to the front end.

[0039] Data lineage: It refers to the record of the source, processing process, and transfer path of data. It details the whole process of data from generation, through various operations such as processing, transformation, and transmission, and finally forming available information or reports.

[0040] Data processing link: The relationships between data processing links include three categories: job and job, job and file, and file and file. The construction of the data link includes the whole link from the data generation source to the data application end. The relationship between jobs is the processing and replication of tables, the relationship between jobs and files is that jobs are loaded and unloaded into files, and the relationship between files and jobs is that files are loaded into tables.

[0041] Configured job processing chain: It is determined according to the execution order of configured jobs in the job configuration information and is used to represent the complete chain of the execution order of configured jobs. Configured jobs are unexecuted jobs that have been configured, and production jobs are jobs that have been executed.

[0042] Figure 1 The following is a flowchart of the job processing broken chain identification method in the embodiments of the present invention, including:

[0043] Step 101, obtain the production job logs in different production environment databases, and store the fields extracted from all production job logs in the database;

[0044] Step 102, parse the SQL statements in the production job logs in the database, and construct data lineage for each SQL statement;

[0045] Step 103: Combine with the ontology knowledge base to perform semantic analysis and correction on the data lineage. The ontology knowledge base is used to store the definitions and relationships of various concepts related to jobs.

[0046] Step 104: Based on all the data lineages, form a data processing link and obtain the SQL execution order corresponding to the data processing link.

[0047] Step 105: Match the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order.

[0048] Step 106: According to the job configuration information, check the production state job execution order to determine the broken link points of the configured state job processing chain, where the configured state job processing chain is determined according to the configured state job execution order in the job configuration information.

[0049] Compared with the conventional broken link identification that requires operation and maintenance personnel to manually check the job configuration, a large number of job execution logs, process records, and business processing rules, resulting in low efficiency, showing problems of insufficient real-time performance, low accuracy, and low automation. The embodiments of the present invention use production state logs to construct the SQL execution order corresponding to the data lineage, that is, the data processing relationship that has been actually and completely processed in production, which has real-time performance, effectiveness, and accuracy, and eliminates the invalidated link processing relationships. Matching the SQL execution order with the fields of the production state job logs to obtain the production state job execution order can cover various database products, and only need to ensure that the database logs can be normally output, and can accurately, quickly, and automatically give a list of broken links in the job processing link. Each step is described in detail below.

[0050] In step 101, obtain the production state job logs in different production environment databases and store the fields extracted from all the production state job logs in the database.

[0051] When collecting the production state job logs, a method of automatically identifying changes in the database environment can be adopted. When a new database is added to the production environment or the type of the existing database changes, the collection strategy can be dynamically adjusted, and it can quickly adapt to the new situation without manual intervention. This can not only improve the timeliness of collecting the production state job logs, but also reduce the operation and maintenance costs. For example, by real-time monitoring information such as the network port changes of the database server and the startup process of the database system, automatically configure the corresponding collection paths and parameters of the production state job logs.

[0052] In an embodiment, storing the fields extracted from all the production state job logs in the database includes:

[0053] Convert production state job logs in different formats into a unified format according to different production environment database types;

[0054] Extract all fields from the production state job logs in the unified format and store them in the database.

[0055] Specifically, introduce artificial intelligence algorithms in the process of converting the production state job log format. Use machine learning models to learn the production state job logs in different formats, so as to achieve more accurate and efficient format conversion. The model can automatically identify and correct format errors according to the semantic information and data characteristics in the production state job logs, and improve the quality of the converted production state job logs. For example, for some production state job logs with non-standard date formats, the model can automatically convert them into a unified standard format.

[0056] A unique hash value can be generated for each production state job log and also stored in the database. The hash value ensures the immutability and integrity of the production state job logs. This makes the basic production state job log information of the production state data lineage more credible and provides a more reliable basis in the face of scenarios such as data auditing and troubleshooting.

[0057] In one embodiment, the fields include one or any combination of database type, database name, executing user, SQL scheduling machine IP address, SQL execution start time, SQL execution end time, and SQL statement.

[0058] In step 102, parse the SQL statements in the production state job logs in the database and construct data lineage for each SQL statement;

[0059] Figure 2 This is the flowchart for constructing data lineage for each SQL statement in the embodiments of the present invention. In one embodiment, parse the SQL statements in the production state job logs in the database and construct data lineage for each SQL statement, including:

[0060] Step 201, split the SQL string in the SQL statement into independent lexical units;

[0061] Specifically, the SQL statements in the database can be preprocessed first, including removing comments, extra spaces and other irrelevant content, and converting them into a clean text form that is convenient for subsequent processing. For example, for the SQL statement "This is a comment\nSELECT*FROM users;" containing multi-line comments, after preprocessing, we get "SELECT*FROM users;", which simplifies the complexity of subsequent operations. Then, the SQL statements are processed as a character stream and recognized and segmented into independent lexical units (Tokens). For example, for "SELECT column1,column2FROM table1 WHERE column1>10;", lexical units such as SELECT, column1, ,, column2, FROM, table1, WHERE, column1, >, 10 will be segmented.

[0062] Step 202: Based on the ontology knowledge base, annotate semantic information for the lexical units;

[0063] The ontology knowledge base can store definitions and relationships of various business vocabulary, SQL syntax elements, and related programming language concepts. Taking a mixed script of SQL and Python as an example, it not only covers the explanations of concepts such as tables, fields, and functions in SQL, but also includes knowledge of Python functions, data types, etc. For example, it differentiates and correlates the semantics and usage scenarios of the SUM function in SQL and the numpy.sum function in Python, so as to help the system better understand and process data processing logics intertwined with different languages.

[0064] If table1 is defined as "user information table" in the ontology knowledge base, then when processing the Token table1, record its semantic information in the ontology knowledge base additionally, which is convenient for subsequent semantic association.

[0065] Step 203: Use regular expressions or finite state automata to generate a sequence of lexical units for all the lexical units;

[0066] Step 204: Construct an abstract syntax tree for each SQL statement according to the sequence of lexical units.

[0067] According to the SQL syntax rules, organize the sequence of lexical units into a tree structure to construct a preliminary Abstract Syntax Tree (AST). For example, for "SELECT column1,column2 FROM table1 JOIN table2 ON table1.id=table2.id;", a tree structure will be constructed with SELECT as the root node, the Projection child node containing column1 and column2, the FROM child node having table1 and the JOIN child node, and the JOIN child node being associated with table2 and the ON condition, etc. Figure 3 This is a schematic diagram of the structure of the Abstract Syntax Tree in an embodiment of the present invention. The nodes in this tree structure are each lexical unit in the sequence of lexical units.

[0068] Figure 4 This is a flowchart for semantic analysis and correction of data lineage in an embodiment of the present invention. In one embodiment, in combination with the ontology knowledge base, semantic analysis and correction of data lineage are performed, including:

[0069] Step 401, verify the syntactic correctness of the Abstract Syntax Tree. The syntactic correctness verification includes clause order verification and structural legality verification;

[0070] Verify the correctness of the syntax in the Abstract Syntax Tree. The correctness verification includes clause order verification (such as FROM before SELECT, GROUP BY after WHERE, etc.) and structural legality verification (such as JOIN must be accompanied by an ON condition, and aggregate functions need to match GROUP BY).

[0071] Step 402, check whether the nodes in the Abstract Syntax Tree are in the database and the ontology database;

[0072] After that, verify the existence of the nodes in the Abstract Syntax Tree, and check whether there are corresponding definitions in the database and the ontology knowledge base for the table names, column names, etc. involved in the AST. For example, if there is "SELECT non_existent_columnFROM table1;" in the AST, confirm through the ontology knowledge base that non_existent_column does not exist, and give corresponding error prompts or correction suggestions.

[0073] Step 403, if so, in combination with the ontology knowledge base, determine the type of each node in the Abstract Syntax Tree;

[0074] Common node types include operator nodes (such as nodes corresponding to keywords like SELECT, JOIN, WHERE, etc.), identifier nodes (table names, column names, aliases, etc.), expression nodes (combinations of operators and operands, such as column1>10), and constant nodes (constant values such as numbers, strings, dates, etc.). For example, for the AST of "SELECT column1,column2 FROM table1 WHERE column1>10;", the SELECT node is labeled as an operator node, column1 and column2 are labeled as identifier nodes, the node corresponding to > is labeled as an operator node, and 10 is labeled as a constant node.

[0075] Step 404, identify the relationships between nodes, where the relationships include at least one of parent-child relationship, sibling relationship, and association relationship;

[0076] Parent-child relationship: Determine the parent and child node relationships for each node. In the AST of SQL, operator nodes are usually the parent nodes of other related nodes. For example, the SELECT node is the parent node of the Projection child node (containing the columns to be selected) and the FROM child node (specifying the data source). Through this parent-child relationship, the structural hierarchy of the query can be clarified.

[0077] Sibling relationship: Identify sibling nodes with the same parent node. For example, in "SELECT column1,column2 FROM table1;", column1 and column2 are sibling nodes under the Projection node, and they are in the same position in the query and are both columns to be selected.

[0078] Association relationship: For nodes involving association operations (such as JOIN), determine the other nodes they are associated with. In "SELECT * FROM table1 JOIN table2 ON table1.id = table2.id;", the JOIN node is associated with table1, table2, and the ON condition node, clarifying the join relationship and join condition between the tables.

[0079] Step 405, based on the ontology knowledge base, analyze the semantic rationality of the relationships between nodes;

[0080] (1) Verify the semantics of the parent - child relationship: According to the definitions of SQL syntax and business semantics in the ontology knowledge base, check whether the parent - child relationship is logical. For example, the ontology knowledge base stipulates that the GROUP BY clause can only appear after the WHERE clause and is a child node of the SELECT operator node. If the position of the GROUP BY node in the AST does not conform to this rule, or the parent node is not a SELECT node, corrections are required.

[0081] (2) Check the consistency of sibling relationships: For sibling nodes, verify whether they are semantically consistent. For example, for sibling column nodes under the Projection node, the ontology knowledge base may stipulate that certain columns can only be queried together, or certain combinations of columns have specific business meanings. If combinations of sibling nodes that do not conform to these regulations are found, the node structure needs to be adjusted or additional semantic constraint information needs to be added.

[0082] (3) Confirm the accuracy of the association relationship: In terms of the association relationship, according to the definitions of the relationships between tables in the ontology knowledge base, check the rationality of the JOIN operation. For example, if the ontology knowledge base defines that table1 and table2 are in a one - to - one association through the id field, and the association condition of the JOIN node in the AST does not conform to this definition, the association condition node needs to be corrected or the association method needs to be adjusted.

[0083] Step 406: Adjust the abstract syntax tree according to the analysis results of semantic rationality;

[0084] The specific adjustment steps are as follows:

[0085] (1) Adjust the node position: When a node position error is found, move the node to the correct position. For example, if the position of the GROUP BY node is incorrect, move it to the correct position after the WHERE clause and under the SELECT node.

[0086] (2) Add or delete nodes: When a node is missing, add a new node. For example, if the ontology knowledge base stipulates that a certain query requires a specific filtering condition, but the AST does not contain the node corresponding to this condition, add the corresponding WHERE clause node and its conditional expression sub - nodes. Conversely, if unnecessary or incorrect nodes are found, such as redundant JOIN nodes or incorrect column reference nodes, delete these nodes.

[0087] (3) Update node attributes and relationships: Update the attributes of nodes to accurately reflect the semantics. For example, if it is found in semantic analysis that the alias of a certain column is used improperly, update the alias attribute according to the definition of the ontology knowledge base. At the same time, adjust the relationships between nodes to ensure the accuracy of the relationships. For example, after correcting the JOIN association conditions, update the connection relationships between the JOIN node and the related table nodes and condition nodes.

[0088] In addition, the traditional full - scale SQL statement parsing is inefficient when dealing with a large amount of data. In the embodiments of the present invention, incremental SQL parsing is developed. When a new SQL statement enters the system, only the newly added or modified part is parsed, and the data lineage relationship is updated. This can greatly reduce the parsing workload and improve the real - time performance of data lineage construction. For example, when a part of the SQL statements in a stored procedure is modified, only this part of the statements needs to be parsed, rather than re - parsing all the SQL statements in the entire stored procedure.

[0089] In step 103, according to all the data lineages, form a data processing link and obtain the SQL execution order corresponding to the data processing link;

[0090] Figure 5 FIG. is a flowchart for forming a data processing link in the embodiments of the present invention. In one embodiment, forming a data processing link according to all the data lineages includes:

[0091] Step 501, according to each data lineage, determine the input source, processing operation, and output destination of the data;

[0092] Parse the SQL statement using lexical analysis, syntax analysis, and semantic analysis techniques, similar to the process of constructing an Abstract Syntax Tree (AST). By analyzing, determine the input source (table or view) of the data, the processing operations of the data (such as selecting specific columns, filtering conditions, aggregation operations, etc.), and the output destination of the data (new table, temporary result set, etc.). For example, for "SELECT SUM(column1) AS total FROM table1 GROUP BY column2", it can be known through analysis that table1 is the input source, perform a sum aggregation operation on column1 and group by column2, and the output result is a new result set containing the total column and the column2 grouping.

[0093] Step 502, according to the input sources, processing operations, and output destinations of all the data, form a data processing link.

[0094] Figure 6This is the correspondence between the SQL execution order and the data processing link in the embodiments of the present invention. Among them, the SQL execution order can form data lineage, and the combination of multiple data lineages can obtain the data processing link. Among them, SQL A to SQL H form the SQL execution order, and each SQL statement corresponds to files, tables, etc. Tables A to H are the data processing links formed by the data lineage.

[0095] In step 104, the fields of the production state job logs stored in the database are matched with the SQL execution order to obtain the production state job execution order;

[0096] For example, for the SQL statements included in a certain production state job log record, it is necessary to find its position in the determined SQL execution order. Through this matching, it is possible to know the execution order of the SQL statements corresponding to each production state job in the entire data processing process.

[0097] In one embodiment, the method further includes:

[0098] After storing the fields extracted from all production state job logs in the database, it further includes:

[0099] Storing the resource consumption data and context environment data of the SQL in the database;

[0100] After matching the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order, it further includes:

[0101] Proofreading the production state job execution order through the resource consumption data and context environment data of the SQL.

[0102] Specifically, traditional matching only relies on the SQL statement itself, and more dimensional information can be introduced, such as the resource consumption data of the SQL (CPU, memory, etc.) and the context environment data (the number of concurrent transactions in the database, the load condition at that time). For example, if the CPU usage rate recorded in the log for a certain SQL statement is significantly higher than that of other similar queries, this feature can be used to more accurately locate and proofread during the matching of the execution order, avoiding matching errors caused by similar SQL statements. For SQL statements executed in a high-concurrency transaction environment, their execution order may be affected by other concurrent operations and is different from that in a low-concurrency situation. Using context information can more accurately proofread and thus match the actual execution order.

[0103] In addition, an incremental matching method can be adopted. When new production state job logs or SQL execution sequences are added, only the newly added part is matched with the existing execution sequence, rather than re-matching all the data. By maintaining an index or cache of the already matched data, the matching position of the newly added data can be quickly located, greatly improving the matching efficiency, which is suitable for a production environment with a large and constantly changing amount of data.

[0104] In step 105, according to the job configuration information, check the execution sequence of the production state jobs to determine the break points of the configured state job processing chain.

[0105] Figure 7 This is a flowchart for determining the break points of the configured state job processing chain in the embodiments of the present invention. In one embodiment, according to the job configuration information, check the execution sequence of the production state jobs to determine the break points of the configured state job processing chain, including:

[0106] Step 701, determine the execution sequence of the configured state jobs through the job configuration information, and obtain the dependency relationships between the configured state jobs;

[0107] Step 702, determine the dependency relationships between the production state jobs according to the execution sequence of the production state jobs;

[0108] Step 703, match the configured situation of the dependency relationships between the configured state jobs with the implementation situation of the dependency relationships between the production state jobs;

[0109] Step 704, when the matching fails, identify the break points of the configured state processing link.

[0110] During specific matching, the parsed SQL execution sequence includes the start time and end time of the SQL execution. When the start time of one SQL < the end time of the previous SQL, it is considered that there is a problem with the data processing relationship between the two SQL statements. Reflected in the job, it means that when the previous job has not been completed, this job has already started to execute. Determine the abnormal points caused by the break of the processing link between the two jobs.

[0111] In one embodiment, matching the configured situation of the dependency relationships between the configured state jobs with the implementation situation of the dependency relationships between the production state jobs includes:

[0112] Identify the types of the dependency relationships between the configured state jobs to be matched and the dependency relationships between the production state jobs. The types of the dependency relationships include one or any combination of forward and backward dependency, conditional dependency, resource dependency, and time window dependency;

[0113] Analyze whether the implementation situation of the dependency relationships is consistent with the configured situation of the dependency relationships between the configured state jobs;

[0114] If they are consistent, it is determined that the match is successful; otherwise, it is determined that the match fails.

[0115] Specifically, if it is a sequential dependency, then whether the subsequent job starts to execute only when the previous job has been completed. If so, the match is successful; if it is a resource dependency, then whether the subsequent job starts to execute only after the previous job releases a specific resource. If so, the match is successful; if it is a time window dependency, for example, it is stipulated in the configuration state that job B must start to execute within 1 hour after the previous job A is completed. In step 703, this time window dependency is considered during the matching verification. If job B in the production state starts to execute more than 1 hour after A is completed, it is considered that the dependency relationship fails to match, and the broken link point can be more accurately located.

[0116] In the embodiments of the present invention, a machine learning algorithm can be used to learn the historical configuration state and production state job execution sequence data to construct an anomaly prediction model. Before performing the matching verification in step 703, the model predicts the possible positions and types of broken link points. For example, according to historical data, it is found that during a specific business peak period, several jobs in a certain configuration state job execution sequence are prone to broken links. The model issues an early warning in advance, and the operation and maintenance personnel can take measures in advance, such as adjusting resource allocation, optimizing the execution plan, etc., to avoid the occurrence of actual broken links.

[0117] After identifying the broken link point (step 704), an automatic repair mechanism can be started. According to the type of the broken link point and the dependency relationship of related jobs, an attempt is made to automatically adjust the job execution sequence or reconfigure job parameters to repair the broken link. For example, if the broken link is caused by incorrect parameter configuration of a certain job, resulting in an overly long execution time and affecting the execution of subsequent jobs, the system automatically adjusts the parameters of this job according to the historical successful execution parameter records and restarts the related jobs to restore the normal operation of the data processing link.

[0118] The embodiments of the present invention also propose a device for identifying job processing broken links. The principle is similar to the method for identifying job processing broken links, which will not be elaborated here.

[0119] Figure 8 It is a schematic structural diagram of the device for identifying job processing broken links in the embodiments of the present invention, including:

[0120] A production state job log acquisition module 801, configured to acquire production state job logs in different production environment databases and store the fields extracted from all production state job logs in a database;

[0121] A data lineage parsing module 802, configured to parse the SQL statements of the production state job logs in the database and construct data lineages for each SQL statement;

[0122] The data lineage correction module 803 is used to combine with the ontology knowledge base to perform semantic analysis and correction on the data lineage, and the ontology knowledge base is used to store the definitions and relationships of various concepts related to the jobs;

[0123] The data processing link formation module 804 is used to form a data processing link according to all the data lineages and obtain the SQL execution order corresponding to the data processing link;

[0124] The production state job execution order obtaining module 805 is used to match the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order;

[0125] The broken link point identification module 806 is used to check the production state job execution order according to the job configuration information to determine the broken link points of the configured state job processing chain, where the configured state job processing chain is determined according to the configured state job execution order in the job configuration information.

[0126] In one embodiment, the fields include one or any combination of database type, database name, executing user, SQL scheduling machine IP address, SQL execution start time, SQL execution end time, and SQL statement.

[0127] In one embodiment, the production state job log obtaining module is used for:

[0128] Convert production state job logs in different formats into a unified format according to different production environment database types;

[0129] Extract all fields from the production state job logs in the unified format and store them in the database.

[0130] In one embodiment, the data lineage parsing module is used for:

[0131] Split the SQL string in the SQL statement into independent lexical units;

[0132] Based on the ontology knowledge base, annotate semantic information to the lexical units;

[0133] Use regular expressions or finite state automata to generate a lexical unit sequence for all the lexical units;

[0134] Construct an abstract syntax tree for each SQL statement according to the lexical unit sequence.

[0135] In one embodiment, the data lineage correction module is used for:

[0136] Perform syntax correctness verification on the abstract syntax tree, and the syntax correctness verification includes clause order verification and structural legality verification;

[0137] Check whether the nodes in the abstract syntax tree are in the database and the ontology database;

[0138] If so, in combination with the ontology knowledge base, determine the type of each node in the abstract syntax tree;

[0139] Identify the relationships between nodes, where the relationships include at least one of parent-child relationship, sibling relationship, and association relationship;

[0140] Analyze the semantic rationality of the relationships between nodes based on the ontology knowledge base;

[0141] Adjust the abstract syntax tree according to the analysis result of semantic rationality.

[0142] In one embodiment, the data processing link formation module is used for:

[0143] Determine the input source, processing operation, and output destination of the data according to each data lineage;

[0144] Form a data processing link according to the input source, processing operation, and output destination of all data.

[0145] In one embodiment, the production state job log acquisition module is further used for:

[0146] After storing the fields extracted from all production state job logs in the database, store the resource consumption data and context environment data of SQL in the database;

[0147] The production state job execution order acquisition module is further used for:

[0148] After matching the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order, proofread the production state job execution order through the resource consumption data and context environment data of SQL.

[0149] In one embodiment, the broken link point identification module is used for:

[0150] Determine the configuration state job execution order through the job configuration information and obtain the dependency relationships between the configuration state jobs;

[0151] Determine the dependency relationships between the production state jobs according to the production state job execution order;

[0152] Match the configuration of the dependency relationships between the configuration state jobs with the implementation of the dependency relationships between the production state jobs;

[0153] When the matching fails, determine the point between the two jobs where the matching fails as the broken link point of the job processing link.

[0154] In one embodiment, the broken-link point identification module is configured to:

[0155] Identify the types of dependencies between the configuration-state jobs to be matched and the dependencies between the production-state jobs, where the types of dependencies include one or any combination of forward and backward dependencies, conditional dependencies, resource dependencies, and time-window dependencies;

[0156] Analyze whether the implementation of the dependencies is consistent with the configuration of the dependencies between the configuration-state jobs;

[0157] If they are consistent, determine that the matching is successful; otherwise, determine that the matching fails.

[0158] In summary, the method and device proposed in the embodiments of the present invention have the following beneficial effects:

[0159] Obtain the production-state job logs in different production environment databases, store the fields extracted from all the production-state job logs in the database; parse the SQL statements of the production-state job logs in the database, and construct the data lineage for each SQL statement; combine with the ontology knowledge base to perform semantic analysis and correction on the data lineage, where the ontology knowledge base is used to store the definitions and relationships of various concepts related to jobs; form a data processing link according to all the data lineages, and obtain the SQL execution order corresponding to the data processing link; match the fields of the production-state job logs stored in the database with the SQL execution order to obtain the production-state job execution order; check the production-state job execution order according to the job configuration information to determine the broken-link points of the configuration-state job processing chain, where the configuration-state job processing chain is determined according to the configuration-state job execution order in the job configuration information. Compared with the conventional broken-link identification that requires operation and maintenance personnel to manually check the job configuration, a large number of job execution logs, process records, and business processing rules, resulting in low efficiency, showing problems of insufficient real-time performance, low accuracy, and low automation, the embodiments of the present invention adopt the production-state logs to construct the SQL execution order corresponding to the data lineage, that is, the data processing relationship that has been actually and completely processed in production, which has real-time performance, effectiveness, and accuracy, and eliminates the invalidated link processing relationships. Matching the SQL execution order with the fields of the production-state job logs to obtain the production-state job execution order can cover various database products, and only needs to ensure that the database logs can be normally output, and can accurately, quickly, and automatically give a list of broken links in the job processing link.

[0160] The embodiments of the present invention also provide a computer device, Figure 9Schematic diagram of a computer device in an embodiment of the present invention. The computer device 900 includes a memory 910, a processor 920, and a computer program 930 stored on the memory 910 and executable on the processor 920. When the processor 920 executes the computer program 930, the above-mentioned job processing disconnection recognition method is implemented.

[0161] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned job processing disconnection recognition method is implemented.

[0162] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned job processing disconnection recognition method is implemented.

[0163] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0164] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the specified functions in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0165] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the specified functions in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so as to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 in one block or a plurality of blocks.

[0167] The specific embodiments described above further elaborate on the objective, technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying broken chains in job processing, characterized in that, including: Obtain production state job logs in databases of different production environments, and store the fields extracted from all the production state job logs in a database; Parse the SQL statements in the production state job logs in the database, and construct data lineage for each SQL statement; Combine with an ontology knowledge base to perform semantic analysis and correction on the data lineage, where the ontology knowledge base is used to store the definitions and relationships of various concepts related to jobs; Form a data processing link based on all the data lineage, and obtain the SQL execution order corresponding to the data processing link; Match the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order; Check the production state job execution order according to job configuration information to determine the broken link points of the configured state job processing chain, where the configured state job processing chain is determined according to the configured state job execution order in the job configuration information.

2. The method according to claim 1, characterized in that, The fields include one or any combination of database type, database name, executing user, SQL scheduling machine IP address, SQL execution start time, SQL execution end time, and SQL statement.

3. The method according to claim 1, characterized in that, Storing the fields extracted from all the production state job logs in the database includes: Convert production state job logs in different formats into a unified format according to different production environment database types; Extract all fields from the production state job logs in the unified format and store them in the database.

4. The method according to claim 1, wherein Parsing the SQL statements in the production state job logs in the database and constructing data lineage for each SQL statement includes: Split the SQL string in the SQL statement into independent lexical units; Based on the ontology knowledge base, annotate semantic information to the lexical units; Use regular expressions or finite state automata to generate a lexical unit sequence from all the lexical units; Construct an abstract syntax tree for each SQL statement according to the lexical unit sequence.

5. The method according to claim 1, characterized in that, Combining with the ontology knowledge base to perform semantic analysis and correction on the data lineage includes: Perform syntax correctness verification on the abstract syntax tree, where the syntax correctness verification includes clause order verification and structural legality verification; Check whether the nodes in the abstract syntax tree are in the database and the ontology database; If so, combine with the ontology knowledge base to determine the type of each node in the abstract syntax tree; Identify the relationships between the nodes, where the relationships include at least one of parent-child relationship, sibling relationship, and association relationship; Analyze the semantic rationality of the relationships between the nodes according to the ontology knowledge base; Adjust the abstract syntax tree according to the analysis result of semantic rationality.

6. The method according to claim 1, wherein Forming a data processing link based on all the data lineage includes: Determine the input source, processing operation, and output destination of the data according to each data lineage; Form a data processing link according to the input source, processing operation, and output destination of all the data.

7. The method according to claim 1, wherein After storing the fields extracted from all the production state job logs in the database, it further includes: Store the resource consumption data and context environment data of the SQL in the database; After matching the fields of the production state job logs stored in the database with the SQL execution order to obtain the production state job execution order, it further includes: Proofread the execution order of production-state jobs based on the resource consumption data and context data of SQL.

8. The method according to claim 1, characterized in that Check the execution order of production-state jobs according to the job configuration information, and determine the break points of the configured-state job processing chain, including: Determine the execution order of configured-state jobs through the job configuration information, and obtain the dependency relationships between configured-state jobs; Determine the dependency relationships between production-state jobs according to the execution order of production-state jobs; Match the configured situation of the dependency relationships between configured-state jobs with the implementation situation of the dependency relationships between production-state jobs; When the matching fails, determine the point between the two jobs with the failed matching as the break point of the job processing link.

9. The method according to claim 8, characterized in that, Match the configured situation of the dependency relationships between configured-state jobs with the implementation situation of the dependency relationships between production-state jobs, including: Identify the types of the dependency relationships between the to-be-matched configured-state jobs and the dependency relationships between production-state jobs, where the types of the dependency relationships include one or any combination of front-back dependency, conditional dependency, resource dependency, and time window dependency; Analyze whether the implementation situation of the dependency relationship is consistent with the configured situation of the dependency relationship between configured-state jobs; If they are consistent, determine that the matching is successful; otherwise, determine that the matching fails.

10. An identification device for broken chain in job processing, characterized in that, Including: A production-state job log acquisition module, which is used to acquire production-state job logs in different production environment databases, and store the fields extracted from all production-state job logs in the database; A data lineage parsing module, which is used to parse the SQL statements in the production-state job logs in the database and construct data lineages for each SQL statement; A data lineage correction module, which is used to perform semantic analysis and correction on the data lineage in combination with the ontology knowledge base, and the ontology knowledge base is used to store the definitions and relationships of various concepts related to jobs; A data processing link formation module, which is used to form a data processing link according to all the data lineages and obtain the SQL execution order corresponding to the data processing link; A production-state job execution order acquisition module, which is used to match the fields of the production-state job logs stored in the database with the SQL execution order to obtain the production-state job execution order; A break point identification module, which is used to check the execution order of production-state jobs according to the job configuration information and determine the break points of the configured-state job processing chain, where the configured-state job processing chain is determined according to the execution order of configured-state jobs in the job configuration information.

11. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1 to 9.

Images