Extraction method and device of field data processing logic and storage medium
By analyzing data processing statements, extracting data processing logic and building a blood relationship network, the problem of traditional methods ignoring data processing logic is solved, and in-depth analysis and efficient evaluation of data processing flow and results are achieved.
Patent Information
- Application Number
- CN202411096141.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-13
AI Technical Summary
In the field of data processing technology, traditional methods ignore data processing logic and are difficult to fully understand and evaluate data processing processes and results. Especially in the big data era when the blood relationship of data is complex, it is difficult to effectively sort out and evaluate.
By analyzing the data processing statements of the target table, an abstract syntax tree is constructed, nodes of field type are extracted, and data processing logic is extracted according to preset rules, data processing logic is extracted and stored for each field in the target table, and blood relationship is constructed network.
It realizes in-depth analysis of data processing processes and results, improves the efficiency and accuracy of data processing logic, reduces the time-consuming process of sorting out blood relationships in fields, and provides more comprehensive data evaluation and query support.
Smart Images
Figure CN119129580A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method, a device and a storage medium for extracting field data processing logic. Background Art
[0002] In the field of data processing technology, data kinship refers to the relationship between data in the process of data generation, processing, circulation and extinction, which is similar to the kinship relationship in human society. In the era of big data, as the amount of data continues to increase, data kinship becomes increasingly complex.
[0003] When analyzing data lineage relationships, the traditional approach is to only analyze the direct relationships between data tables and fields, ignoring the data processing logic. Without analyzing the data processing logic, it is difficult to fully understand and evaluate the data processing flow and results. Summary of the invention
[0004] The embodiments of the present application provide a field data processing logic extraction method, device and storage medium for in-depth analysis of data processing processes and results.
[0005] In a first aspect, an embodiment of the present application provides a method for extracting field data processing logic, including:
[0006] By analyzing any data processing statement for constructing the target table, a first abstract syntax tree corresponding to the data processing statement is obtained;
[0007] Determine a node of a field type from the first abstract syntax tree;
[0008] Extracting at least one data processing logic pair in the node from the field type according to a preset rule, wherein the preset rule defines condition information and result information constituting the data processing logic pair;
[0009] Based on the data processing logic pairs of each data processing statement, the data processing logic of each field in the target table is obtained.
[0010] In an embodiment of the present application, the data processing statements for constructing the target table are first parsed to obtain a first abstract syntax tree, and the field data processing logic is subsequently extracted through the first abstract syntax tree, which can more efficiently process the data processing statements and improve the efficiency and accuracy of the field data processing logic. Instead of focusing solely on the logical relationship between tables and tables, fields and fields, the focus is expanded to the field data processing logic level, thereby achieving a comprehensive and in-depth analysis of the data processing flow and results.
[0011] Optionally, the step of analyzing any data processing statement for constructing the target table to obtain a first abstract syntax tree corresponding to the data processing statement includes:
[0012] Parsing any data processing statement for constructing the target table through a data engine to generate a first abstract syntax tree corresponding to the data processing statement;
[0013] The first abstract syntax tree is obtained from the data engine through a plug-in mechanism.
[0014] In the embodiment of the present application, the data processing statement is parsed by the data engine to obtain the first abstract syntax tree, which lays the foundation for the subsequent extraction of field data processing logic and ensures that query requirements from different users can be processed. The first abstract syntax tree can be captured without modifying the source code through the plug-in of the data engine, providing support for the subsequent extraction of field data processing logic.
[0015] Optionally, after obtaining the first abstract syntax tree corresponding to the data processing statement and before determining a node of a field type from the first abstract syntax tree, the method further includes:
[0016] After serializing the first abstract syntax tree, the serialization is asynchronously sent to a message queue;
[0017] After obtaining the serialized first abstract syntax tree from the message queue, converting it into the first abstract syntax tree of the object class.
[0018] In the embodiment of the present application, by serializing the first abstract syntax tree, it can be converted into a format that is more suitable for network transmission, so as to facilitate the transmission of the first abstract syntax tree between different systems.
[0019] Optionally, the structure of the preset rule consists of a non-terminal symbol and a substitute;
[0020] The step of extracting at least one data processing logic pair from the node according to a preset rule includes:
[0021] Extracting information corresponding to the keyword indicated by the substitute from the node using the keyword indicated by the substitute in the preset rule;
[0022] The information corresponding to the keyword representing the data processing logic pair is determined as a data processing logic pair.
[0023] In the embodiment of the present application, the field data processing logic is extracted according to the preset rules, making the extraction more convenient and laying the foundation for the subsequent construction of the blood relationship network. The corresponding information is extracted according to the keywords in the preset rules, and the storage field data processing logic is processed according to the data processing logic to facilitate subsequent queries.
[0024] Optionally, a predefined entity class is used to extract the field data processing logic and store the field data processing logic according to the predefined entity class; a storage structure of a predefined abstract syntax tree is used to store the first abstract syntax tree and the second abstract syntax tree according to the storage structure of the predefined abstract syntax tree.
[0025] In the embodiment of the present application, the predefined entity class ensures that the key information of the table name and field name of the first abstract syntax tree can be accurately extracted, providing a basis for subsequent extraction and query optimization. Preparing a suitable storage structure to store the first abstract syntax tree and the second abstract syntax tree can improve the efficiency of subsequent traversal and node recognition of the first abstract syntax tree, so that the abstract syntax tree can be processed efficiently and in large quantities.
[0026] Optionally, extracting information corresponding to the keyword indicated by the substitute from the node includes:
[0027] According to the keyword indicated by the substitute of any one of the preset rules, information corresponding to the keyword indicated by the substitute is extracted from the node, and the extracted information is stored in the node of the second abstract syntax tree.
[0028] In the embodiment of the present application, the extracted information is stored in the second abstract syntax tree, and each node of the second abstract syntax tree corresponds to a rule, which can clearly show the extracted field data processing logic.
[0029] Optionally, after obtaining the data processing logic of each field in the target table, the method further includes:
[0030] A blood relationship network is constructed through the data processing logic of each field in the target table.
[0031] In a second aspect, an embodiment of the present application provides a device for extracting field data processing logic, including:
[0032] An analysis module, configured to obtain a first abstract syntax tree corresponding to any data processing statement for constructing a target table by analyzing the data processing statement;
[0033] A determination module, configured to determine a node of a field type from the first abstract syntax tree;
[0034] An extraction module, used to extract at least one data processing logic pair in the node from the field type according to a preset rule, wherein the preset rule defines condition information and result information constituting the data processing logic pair;
[0035] The processing module is used to obtain the data processing logic of each field in the target table based on the data processing logic pair of each data processing statement.
[0036] Optionally, the analysis module is specifically used for:
[0037] Parsing any data processing statement for constructing the target table through a data engine to generate a first abstract syntax tree corresponding to the data processing statement;
[0038] The first abstract syntax tree is obtained from the data engine through a plug-in mechanism.
[0039] Optionally, the determining module is further used for:
[0040] After serializing the first abstract syntax tree, the serialization is asynchronously sent to a message queue;
[0041] After obtaining the serialized first abstract syntax tree from the message queue, converting it into the first abstract syntax tree of the object class.
[0042] Optionally, the extraction module is specifically used for:
[0043] Extracting information corresponding to the keyword indicated by the substitute from the node using the keyword indicated by the substitute in the preset rule;
[0044] The information corresponding to the keyword representing the data processing logic pair is determined as a data processing logic pair.
[0045] Optionally, a predefined entity class is used to extract the field data processing logic and store the field data processing logic according to the predefined entity class; a storage structure of a predefined abstract syntax tree is used to store the first abstract syntax tree and the second abstract syntax tree according to the storage structure of the predefined abstract syntax tree.
[0046] Optionally, the extraction module is specifically used for:
[0047] According to the keyword indicated by the substitute of any one of the preset rules, information corresponding to the keyword indicated by the substitute is extracted from the node, and the extracted information is stored in the node of the second abstract syntax tree.
[0048] Optionally, the processing module is further used for:
[0049] A blood relationship network is constructed through the data processing logic of each field in the target table.
[0050] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.
[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, wherein when the program is run on the computer device, the computer device executes the steps of the above method.
[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned master-slave machine abnormal recovery method. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 A system architecture diagram provided for an embodiment of the present application;
[0055] Figure 2 A schematic diagram of a field data processing logic method flow provided in an embodiment of the present application Figure 1 ;
[0056] Figure 3 A schematic diagram of a field data processing logic method flow provided in an embodiment of the present application Figure 2 ;
[0057] Figure 4 A schematic diagram of the structure of a field data processing logic device provided in an embodiment of the present application;
[0058] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0060] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.
[0061] Data kinship. Data kinship refers to the relationship between data in the process of data generation, processing, circulation and extinction, which is similar to the kinship in human society. Data kinship is generated by data processing logic.
[0062] Abstract syntax tree (AST for short) is an abstract representation of the syntax structure of source code. It represents the syntax structure of programming language in the form of a tree, and each node in the tree represents a structure in the source code.
[0063] To facilitate understanding of this solution, the application scenario of this solution is introduced below.
[0064] When sorting out data lineage relationships, the target table is often processed from multiple source tables. When it is necessary to analyze the source of the data composition of the target table, we can generally only know which source table the target table is from, or which field in the source table the fields in the target table are obtained from, but we cannot know how the fields in the target table are obtained through the fields in the source table and what kind of field data processing logic is used. Therefore, the complexity and diversity of the field processing logic are ignored. This application not only focuses on the relationship between the target table and the source table, and the relationship between the target table and the fields in the source table, but also pays more attention to the field data processing logic in the target table, so that a more comprehensive and in-depth evaluation of the data can be performed, especially in big data testing, which can reduce the time spent on sorting out field lineage relationships. The specific implementation steps of this application are introduced below:
[0065] First, the present application predefines the storage structure of the entity class and the abstract syntax tree. The entity class is used to store the data processing logic, and the abstract syntax tree is used to store the first abstract syntax tree and the second abstract syntax tree. Then, the data processing statement is received, the data processing statement for constructing the target table is analyzed one by one, and the data processing statement is parsed by the data engine, and the parsed data processing statement is generated into a first abstract syntax tree. Then the first abstract syntax tree is serialized and sent to the message queue asynchronously, and the serialized first abstract syntax tree is converted into the first abstract syntax tree of the object class. Then the first abstract syntax tree of the object class is traversed, the node of the field type is determined from the first abstract syntax tree, and the data processing logic pair in the node of the field type is extracted according to the preset rules. In the embodiment of the present application, the preset rules are constructed according to the BNF paradigm rules, and the BNF paradigm is a definition specification that uses recursive ideas to express a set of computer language symbols. It describes the grammatical structure of the language through a series of production rules, and can accurately describe the structure and semantics of the field processing logic. In the preset rules, the condition information and result information of the data processing logic are defined, and the condition information and result information are added to the data processing logic structure. Finally, according to the pre-defined entity class, the data processing logic is extracted into the entity class to complete the extraction of the data processing logic. After the data processing logic is extracted, the corresponding data lineage relationship is also clear at a glance.
[0066] See also Figure 1 , is a system architecture diagram provided in an embodiment of the present application, the system architecture includes a terminal device 101 and a server 102.
[0067] The terminal device 101 is pre-installed with a service application for text matching, wherein the service application is a client application, a web application, a small program application, etc. The terminal device 101 can be a smart phone, a POS machine, etc., but is not limited thereto.
[0068] Server 102 is the backend server for business applications. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.
[0069] See also Figure 2 , a method for extracting field data processing logic provided in an embodiment of the present application, comprising the following steps:
[0070] Step 201: Analyze any data processing statement for constructing a target table to obtain a first abstract syntax tree corresponding to the data processing statement.
[0071] Specifically, the target table is a table obtained through data processing, and the source table is another table constituting the target table. The target table can be obtained through data processing from one source table or from multiple source tables. When constructing the target table, the source table can be constructed into the target table through data processing statements, and the data processing statements can be analyzed to obtain the first abstract syntax tree of the data processing statements.
[0072] For example, the data processing statement is an SQL query statement, which includes the operation of aliasing and filtering the fields of a data table, as shown in Case 1:
[0073] cont.id AS cont_id,
[0074] cont.buss_typ AS business_type,
[0075] cont.dscnt_typ AS disc_type,
[0076] cont.dscnt_dt AS disc_date,
[0077] list.drft_id AS bill_id,
[0078] list.int_rate AS disc_rate,
[0079]
[0080] From the above SQL statements, we can see that in the target table, the field cont.id in the source table is used as cont_id in the target table, the field cont.buss_typ is used as business_type in the target table, the field cont.dscnt_typ is used as disc_type in the target table, the field cont.dscnt_dt is used as disc_date in the target table, and the field list.drft_id is used as bill_id in the target table. By analyzing the above data processing statements, we get the first abstract syntax tree. As shown in Case 2, the first abstract syntax tree contains all the data processing statements:
[0081]
[0082] CaseExpression
[0083] CaseCondition(when cont2.idis not null)
[0084] CaseResult(then'Y')
[0085] CaseElse
[0086] CaseResult(then'N')
[0087] EndCase
[0088] Alias(first_disc_flag) / / Add alias to case expression
[0089] FromClause
[0090] TableReference
[0091] SubQuery
[0092] SelectClause(Select*) / / List the required columns more specifically as needed
[0093] FromClause
[0094] TableName(table_name) / / You can add other clauses such as whereClause for the table
[0095] EndSubQuery
[0096] Alisa(cont)
[0097] In some embodiments, by analyzing any data processing statement for constructing a target table, a first abstract syntax tree corresponding to the data processing statement is obtained, including:
[0098] The data engine is used to parse any data processing statement for constructing the target table to generate a first abstract syntax tree corresponding to the data processing statement; and the first abstract syntax tree is obtained from the data engine through a plug-in mechanism.
[0099] Specifically, before generating the first abstract syntax tree, the data processing statement needs to be parsed first, and then the parsed data processing statement is captured through a plug-in mechanism, so as to generate the first abstract syntax tree.
[0100] For example, when parsing data processing statements, a Hive engine may be used. The Hive engine receives tasks such as SQL queries, and parses data processing statements of tasks such as SQL queries to generate a first abstract syntax tree. The Hive engine is a Hadoop-based data warehouse tool that allows big data to be processed in a Hadoop cluster using a SQL-like language.
[0101] After receiving tasks such as SQL queries, the Hive engine performs lexical analysis on the data processing statements of tasks such as SQL queries, and decomposes the data processing statements of tasks such as SQL queries into a series of tags, including keywords, identifiers, operation types, etc., which are the basic elements for constructing data processing statements of tasks such as SQL queries. Through lexical analysis, these tags can be accurately identified, laying the foundation for subsequent syntax analysis, and can deeply understand the intention of SQL data processing statements.
[0102] Since different data engines parse data processing statements to obtain different first abstract syntax trees, it is necessary to predefine the tags, that is, to specify which tags refer to fields, which tags refer to tables, keywords or operation types, etc.
[0103] By using the syntax rules of the SQL language, the tokens obtained by the lexical analysis are combined into nodes of the first abstract syntax tree, thereby constructing the first abstract syntax tree. The first abstract syntax tree can accurately reflect the syntax structure of the SQL data processing statement.
[0104] After obtaining the first abstract syntax tree, the first abstract syntax tree is captured through Hive Hook. Hive Hook is a mechanism for intercepting events, messages, or function calls during processing. Hive Hook is bound to the internal working mechanism of Hive and does not require recompiling Hive. Hive Hook provides the ability to use Hive to extend and integrate external functions. Using Hive Hook to capture the first abstract syntax tree allows the first abstract syntax tree data to be captured without modifying the Hive source code, which facilitates subsequent processing.
[0105] In some embodiments, after obtaining the first abstract syntax tree corresponding to the data processing statement and before determining the node of the field type from the first abstract syntax tree, it also includes: serializing the first abstract syntax tree and sending it asynchronously to the message queue; after obtaining the serialized first abstract syntax tree from the message queue, converting it into the first abstract syntax tree of the object class.
[0106] Specifically, the first abstract syntax tree is serialized, that is, the first abstract syntax tree is converted into a format suitable for network transmission, such as JSON format, to ensure that the first abstract syntax tree can be transmitted and exchanged between different systems. By serializing the first abstract syntax tree, the first abstract syntax tree can also be asynchronously sent to a Kafka (referred to as Kafka) message queue for bypass parsing and extraction, thereby realizing asynchronous processing and distribution of the first abstract syntax tree, so that a large number of first abstract syntax trees can be received and processed in real time.
[0107] After receiving the Kafka message, the serialization framework Jackson library is used to convert the first abstract syntax tree in JSON format back to the object class first abstract syntax tree. A load balancing algorithm (minimum number of connections) is used to determine which server node receives the current first abstract syntax tree, and the first abstract syntax tree is forwarded to the selected server node through the internal communication protocol (gRPC).
[0108] Step 202: Determine a node of a field type from the first abstract syntax tree.
[0109] Specifically, after obtaining the first abstract syntax tree, before traversing all the nodes of the first abstract syntax tree (the first abstract syntax tree includes a root node and child nodes), the node types of all nodes are detected. The present application predetermines that different nodes represent different meanings. For example, in Case 2, SelectItem represents a node of field type, and TableReference represents a "table". Different processing methods are used for different nodes. After determining the type of the node, the attribute information of each node is recorded. The attribute information of the node includes the operation type of the node, the list of child nodes, the associated table name, the field name, etc. Among them, the operation type represents the intention and purpose of the data processing statement. For example, SELECT (query), UPDATE (update), and INSERT (insert) in the SQL query statement are different operation types. Therefore, when recording the node attribute information, the operation type of the node in the attribute information must also be identified and stored. Detecting the node type and recording the operation type of the identified node attribute information can improve the efficiency of node traversal, and deeply understand the structure and content of the first abstract syntax tree, which provides convenience for subsequent data processing logic extraction and query.
[0110] Starting from the root node of the first abstract syntax tree, the first abstract syntax tree may be traversed using a depth-first search algorithm.
[0111] Step 203: extract at least one data processing logic pair from the node of the field type according to a preset rule, wherein the preset rule defines condition information and result information constituting the data processing logic pair. The structure of the preset rule consists of a non-terminal symbol and a substitute.
[0112] Specifically, after determining the operation type of the node of the first abstract syntax tree, the nodes of the first abstract syntax tree are traversed to extract the field data processing logic. The field data processing logic usually includes key information such as field name, data type, and table to which it belongs. In the process of traversing the nodes, it is necessary to identify which nodes are field type nodes. Field type nodes usually represent tables and fields referenced in data processing statements. By identifying field type nodes, the field information involved in the data processing statement can be accurately extracted, providing a basis for subsequent field data processing logic extraction and query optimization. Usually, whether a node is a field type node is determined based on the node type and attribute information.
[0113] After determining the node of the field type, extract the data processing logic pair in the node of the field type according to the preset rules. The preset rules are matching rules pre-set in the embodiment of the present application. The field data processing logic of the node of the field type is matched with the preset rules. If the field data processing logic in the node of the field type meets the preset rules, it is extracted according to the preset rules; if it does not meet the preset rules, it is not extracted. At the same time, recording the data processing logic that succeeds or fails in matching can provide a basis for subsequent exception handling and error troubleshooting.
[0114] The preset rules are set according to the BNF paradigm rules. The basic structure of the BNF paradigm consists of non-terminals and replacements in the form of::=. Among them, non-terminals refer to symbols that can be further explained and defined by the replacements on the right, and are enclosed in angle brackets. Replacements refer to symbols that can be used to replace non-terminals, which can be terminals or non-terminals, separated by spaces or vertical lines. Non-terminals refer to symbols that have not yet been defined, that is, symbols of certain abstract concepts in the language that cannot appear directly in the language. Terminal: refers to symbols that can no longer be replaced, usually letters, numbers, operators, etc. Case three lists the simplified BNF paradigm expression of case when:
[0115] <case_experession> ::="CASE"<when_clause_list><else_clause> "END"
[0116] <when_clause_list> ::=<when_clause> |<when_clause><when_clause_list>
[0117] <when_clause> ::=“WHEN” <condition>"THEN” <result>
[0118] <else_clause>::="ELSE” <result>
[0119] <condition> ::＝ <experssion>
[0120] <result> ::＝ <experssion>
[0121] <experssion> ::＝ <literal>|<column_reference> |<function_call> .
[0122] In some embodiments, according to preset rules, at least one data processing logic pair in a node is extracted, including: extracting information corresponding to the keyword indicated by the substitute from the node through the keyword indicated by the substitute in the preset rule; and determining the information corresponding to the keyword representing the data processing logic pair as a data processing logic pair.
[0123] Specifically, from the above example, it can be seen that the keywords indicated by the substitute in the preset rule are "CASE" and "END". According to the preset rule, a recursive descent parser is used to<case_expression> The rule starts trying to match<case_expression> , first matching the "CASE" keyword, then recursively matching<when_clause_list> , followed by an optional<else_clause> , and finally matches "END".
[0124] Parse possible nested functions or subqueries to ensure the integrity and accuracy of data processing logic.<when_clause_list> , looping through each<when_clause> , each<when_clause> By matching the following <condition>, such as "cont2.id is not null", followed by "THEN" <result>, such as "Y" in Case 2.<else_clause> By matching the following "ELSE" <result>, as in Case 2, "N".
[0125] for <condition>and <result>, is defined as <expression>expression type, which means that further <expression>Matching rules will involve further analysis of text, function calls, etc. For example, for the above case 2, <experssion> ::＝ <literal>|<column_reference> |<function_call> The BNF expression of " can be expanded as shown in Example 4:
[0126] <literal> ::＝ <stringliteral> | <nullliteral>
[0127] <stringliteral> ::＝" <stringcharacters> ”
[0128] <stringcharacters> ::＝" <stringcharacter> ”+
[0129] <stringcharacter>:: = / * Any character except double quotes, or escaped double quotes * /
[0130] <nullliteral>::="NULL".
[0131] When "NULL" appears, it is considered that it cannot be replaced and the matching stops. "NULL" is the terminator.
[0132] Extract the conditional expression part from the case node. For each WHEN condition and the corresponding THEN result encountered, treat it as a data processing logic pair. When encountering an ELSE node, use the ELSE node result as the default case.
[0133] As shown in Case 5, the data processing logic is extracted as follows:
[0134] Case_logic={
[0135]
[0136] According to the data processing logic, the above cases are extracted, and the case when extraction results are shown in Case 6:
[0137]
[0138] Step 204: Based on the data processing logic pairs of each data processing statement, obtain the data processing logic of each field in the target table.
[0139] Specifically, after the data processing logic pairs of each data processing statement are determined, the data processing logic pairs are extracted to obtain the data processing logic in the target table.
[0140] In some embodiments, the preset rules include: a first rule characterizing a sentence structure having data processing logic; a second rule characterizing the data processing logic in each part of the sentence structure; and a third rule for the expression structure of the data processing logic in each part.
[0141] Specifically, as shown in the above case 3:
[0142] <case_experession> ::="CASE"<when_clause_list><else_clause> "END" is the first rule that represents the sentence structure with data processing logic;
[0143] against<when_clause_list> ,<when_clause> ,<else_clause> These three parts,
[0144] <when_clause_list> ::=<when_clause> |<when_clause><when_clause_list> ,
[0145] <when_clause>::="WHEN” <condition>"THEN” <result>,
[0146] as well as<else_clause> ::="ELSE" <result>The second rule characterizes the logic of data processing in each part;
[0147] against<when_clause> "WHEN" in <condition>"THEN” <result>, which is the third rule for the expression structure of the data processing logic in each part.
[0148] In some embodiments, extracting information corresponding to a keyword indicated by a substitute from a node includes: extracting information corresponding to the keyword indicated by a substitute from a node according to the keyword indicated by a substitute according to any one of preset rules, and storing the extracted information in a node of a second abstract syntax tree.
[0149] Specifically, in the preset rule, the information corresponding to the keyword indicated by the substitute is extracted, and the second abstract syntax tree is constructed according to the information corresponding to the keyword. The second abstract syntax tree includes multiple nodes, each node is a field data processing logic, and each field data processing logic corresponds to a rule.<case_experession> ::="CASE"<when_clause_list><else_clause> The first rule corresponding to "END" is a field data processing logic.<when_clause_list> ::=<when_clause> |<when_clause><when_clause_list> ,<when_clause> ::="WHEN" <condition>"THEN” <result>、<else_clause>::="ELSE” <result>The corresponding second rule is a field data processing logic, "WHEN" <condition>"THEN” <result>The corresponding third rule is a field data processing logic.
[0150] In some embodiments, after obtaining the data processing logic of each field in the target table, it also includes: constructing a blood relationship network through the data processing logic of each field in the target table.
[0151] Specifically, all the extracted data processing logic and other relevant information are summarized and organized. Ensure data integrity and consistency. Through summarization and organization, a clear and accurate data set is generated to provide strong data guarantee for the subsequent construction and storage of blood relationships. In the construction and storage of blood relationship networks, the embodiment of the present application adopts a graph database (such as Neo4j) as a storage medium. By storing the nodes of the aggregated entity classes in a graph database, an intuitive and clear blood relationship network can be constructed. Through the graph database, users can easily query and understand the dependencies between data, providing strong support for big data governance and analysis. In the graph database, the data processing logic relationship between data can be quickly queried and extracted to form a blood chain. This is achieved by executing a graph query, and all paths from one node to another can be found.
[0152] In some embodiments, a predefined entity class is used to extract the field data processing logic and store the field data processing logic according to the predefined entity class; a storage structure of a predefined abstract syntax tree is used to store the first abstract syntax tree and the second abstract syntax tree according to the storage structure of the predefined abstract syntax tree.
[0153] Specifically, before traversing the first abstract syntax tree, it is also necessary to set up a configuration file. The configuration file specifies the entity class used to store the field data processing logic. The entity class is used to specify the source information of the field, such as which tables the field comes from, what the field data processing logic is, etc. By predefining the entity class, the subsequent extraction of the field data processing logic is facilitated.
[0154] The configuration file also specifies a storage structure for storing the first abstract syntax tree and the second abstract syntax tree. After loading the predefined entity class, the storage structure of the predefined abstract syntax tree is initialized, and the structure is used to store information of each node in the first abstract syntax tree and the second abstract syntax tree, such as node type, node attribute, child node, etc. Preparing a suitable storage structure for the abstract syntax tree can improve the efficiency of traversing the abstract syntax tree and identifying nodes.
[0155] In order to further explain the present application, the following describes in detail the method for extracting field data processing logic using SQL query as a specific data processing statement and CASE node identification as a specific application scenario, including the following steps: Figure 3 As shown:
[0156] Step 301: predefine an entity class, and identify the table names and field names of the first abstract syntax tree and the second abstract syntax tree according to the predefined entity class. The entity class represents the data structure of the table or field.
[0157] Step 302: predefine a data structure of node information of an abstract syntax tree, and store the node information of the first abstract syntax tree and the second abstract syntax tree according to the predefined data structure of node information of an abstract syntax tree.
[0158] Step 303: Obtain SQL query statements through the Hive engine.
[0159] Step 304: perform lexical analysis on the SQL query statement, and decompose the SQL query statement into a series of tokens, including keywords, identifiers, operators, etc. of the SQL query statement.
[0160] Step 305: Using the syntax rules of the SQL language, combine the tokens obtained by the lexical analysis into a first abstract syntax tree.
[0161] Step 306: Capture the first abstract syntax tree through Hive Hook.
[0162] Step 307: Serialize the first abstract syntax tree into JSON format and asynchronously transmit it to the server node through the Kafka message queue.
[0163] Step 308: Determine the node of the receiving server according to the load balancing strategy.
[0164] Step 309: traverse the nodes of the first abstract syntax tree and check the type of each node.
[0165] Step 310: Record the attribute information of each node of the first abstract syntax tree, including the operation type of the node, the associated table name, the field name, etc.
[0166] Step 311: Identify the operation type of each node of the first abstract syntax tree.
[0167] Step 312: traverse each node of the first abstract syntax tree and determine a field node from the node.
[0168] Step 313: Obtain field information of the field node, including field name, alias, and association relationship between fields.
[0169] Step 314: Match the field nodes according to the preset rules. For CaseExpression, match the keyword CASE, then match the when clause list, then match the else clause, and finally match END.
[0170] Step 315: construct a second abstract syntax tree using the node information of the field nodes that conform to the preset rule, wherein each node of the second abstract syntax tree corresponds to a rule.
[0171] Step 316: According to the second abstract syntax tree, determine each WHEN condition information and corresponding THEN result information from the CASE node, and store them as a data processing logic pair.
[0172] Step 317: Extract the data processing logic to obtain the data processing logic of each field in the target table.
[0173] Step 318: Summarize the data processing logic of all fields to obtain the blood relationship network of each field in the target table.
[0174] Based on the same technical concept, the embodiment of the present application provides a device for extracting field data processing logic, such as Figure 4 As shown, the device 400 includes:
[0175] An analysis module 401 is used to analyze any data processing statement for constructing a target table to obtain a first abstract syntax tree corresponding to the data processing statement;
[0176] A determination module 402, configured to determine a node of a field type from the first abstract syntax tree;
[0177] An extraction module 403 is used to extract at least one data processing logic pair in the node from the field type according to a preset rule, wherein the preset rule defines condition information and result information constituting the data processing logic pair;
[0178] The processing module 404 is used to obtain the data processing logic of each field in the target table based on the data processing logic pair of each data processing statement.
[0179] Optionally, the analysis module 401 is specifically used for:
[0180] Parsing any data processing statement for constructing the target table through a data engine to generate a first abstract syntax tree corresponding to the data processing statement;
[0181] The first abstract syntax tree is obtained from the data engine through a plug-in mechanism.
[0182] Optionally, the determining module 402 is further configured to:
[0183] After serializing the first abstract syntax tree, the serialization is asynchronously sent to a message queue;
[0184] After obtaining the serialized first abstract syntax tree from the message queue, converting it into the first abstract syntax tree of the object class.
[0185] Optionally, the extraction module 403 is specifically used for:
[0186] Extracting information corresponding to the keyword indicated by the substitute from the node using the keyword indicated by the substitute in the preset rule;
[0187] The information corresponding to the keyword representing the data processing logic pair is determined as a data processing logic pair.
[0188] Optionally, a predefined entity class is used to extract the field data processing logic and store the field data processing logic according to the predefined entity class; a storage structure of a predefined abstract syntax tree is used to store the first abstract syntax tree and the second abstract syntax tree according to the storage structure of the predefined abstract syntax tree.
[0189] Optionally, the extraction module 403 is specifically used for:
[0190] According to the keyword indicated by the substitute of any one of the preset rules, information corresponding to the keyword indicated by the substitute is extracted from the node, and the extracted information is stored in the node of the second abstract syntax tree.
[0191] Optionally, the processing module 404 is further used for:
[0192] A blood relationship network is constructed through the data processing logic of each field in the target table.
[0193] Based on the same technical concept, the embodiment of the present application provides a computer device, which may be a terminal or a server, such as Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 For example, the processor 501 and the memory 502 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.
[0194] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the steps included in the above-mentioned field data processing logic method by executing the instructions stored in the memory 502.
[0195] The processor 501 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, by running or executing instructions stored in the memory 502 and calling data stored in the memory 502. Optionally, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.
[0196] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0197] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0198] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned field data processing logic method.
[0199] Based on the same inventive concept, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned field data processing logic method.
[0200] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0201] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0202] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0204] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.< / result> < / condition> < / result> < / result> < / condition> < / result> < / condition> < / result> < / result> < / condition> < / nullliteral> < / stringcharacter> < / stringcharacter> < / stringcharacters> < / stringcharacters> < / stringliteral> < / nullliteral> < / stringliteral> < / literal> < / literal> < / experssion> < / expression> < / expression> < / result> < / condition> < / result> < / result> < / condition> < / literal> < / experssion> < / experssion> < / result> < / experssion> < / condition> < / result> < / result> < / condition>
Claims
1. A method for extracting field data processing logic, characterized in that: include: By analyzing any data processing statement for constructing the target table, a first abstract syntax tree corresponding to the data processing statement is obtained; Determine a node of a field type from the first abstract syntax tree; Extracting at least one data processing logic pair in the node from the field type according to a preset rule, wherein the preset rule defines condition information and result information constituting the data processing logic pair; Based on the data processing logic pairs of each data processing statement, the data processing logic of each field in the target table is obtained.
2. The method according to claim 1, characterized in that The step of analyzing any data processing statement for constructing the target table to obtain a first abstract syntax tree corresponding to the data processing statement includes: Parsing any data processing statement for constructing the target table through a data engine to generate a first abstract syntax tree corresponding to the data processing statement; The first abstract syntax tree is obtained from the data engine through a plug-in mechanism.
3. The method according to claim 2, characterized in that After obtaining the first abstract syntax tree corresponding to the data processing statement and before determining the node of the field type from the first abstract syntax tree, the method further includes: After serializing the first abstract syntax tree, the serialization is asynchronously sent to a message queue; After obtaining the serialized first abstract syntax tree from the message queue, converting it into the first abstract syntax tree of the object class.
4. The method according to claim 1, characterized in that The structure of the preset rule consists of non-terminal symbols and substitutes; The step of extracting at least one data processing logic pair from the node according to a preset rule includes: Extracting information corresponding to the keyword indicated by the substitute from the node using the keyword indicated by the substitute in the preset rule; The information corresponding to the keyword representing the data processing logic pair is determined as a data processing logic pair.
5. The method according to claim 4, characterized in that Extracting information corresponding to the keyword indicated by the substitute from the node includes: According to the keyword indicated by the substitute of any one of the preset rules, information corresponding to the keyword indicated by the substitute is extracted from the node, and the extracted information is stored in the node of the second abstract syntax tree.
6. The method according to any one of claims 1 to 5, characterized in that: Predefined entity class, after extracting the field data processing logic, storing the field data processing logic according to the predefined entity class; A storage structure of an abstract syntax tree is predefined, and the first abstract syntax tree and the second abstract syntax tree are stored according to the storage structure of the predefined abstract syntax tree.
7. The method according to any one of claims 1 to 5, characterized in that: After obtaining the data processing logic of each field in the target table, it also includes: A blood relationship network is constructed through the data processing logic of each field in the target table.
8. A device for extracting field data processing logic, characterized in that: include: An analysis module, configured to obtain a first abstract syntax tree corresponding to any data processing statement for constructing a target table by analyzing the data processing statement; A determination module, configured to determine a node of a field type from the first abstract syntax tree; An extraction module, used to extract at least one data processing logic pair in the node from the field type according to a preset rule, wherein the preset rule defines condition information and result information constituting the data processing logic pair; The processing module is used to obtain the data processing logic of each field in the target table based on the data processing logic pair of each data processing statement.
9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.