ANTLR 4-based data processing architecture method

Abstract

The invention discloses a data processing architecture method based on ANTLR 4, which supports richer rule grammar, provides a plurality of preset data calculation rules, and integrates a hot loading mechanism for periodically updating data processing rules. A technical scheme with flexible configuration, compilation-free and hot loading can be provided for processing logic arrangement of real-time and offline big data; in the big data processing process, after the data processing logic is temporarily modified, the rule can take effect timely and conveniently without recompiling and deploying or restarting a running program, and the processing result of the latest data processing logic is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a data processing architecture method based on ANTLR 4. Background Art

[0002] Big data refers to a group of data whose size exceeds the ability of traditional database software tools to capture, store, manage and analyze. Therefore, the processing of big data often requires the use of distributed multiple computer resources, and the technology related to the coordinated use of distributed computing resources for big data processing is big data processing technology. The current mainstream big data processing frameworks and technologies include HADOOP, SPARK, FLINK, STORM, HIVE, HBASE and KAFKA. These frameworks or technology platforms can coordinate the use of distributed computer resources for big data analysis, processing and storage operations. Big data developers can use the above big data processing frameworks and platforms to build big data processing applications, thereby realizing the processing and analysis of massive data.

[0003] Syntactic analysis is a process of analyzing an input text (such as program code) consisting of a sequence of words according to a given formal grammar and determining its grammatical structure. A parser usually appears as a component of a compiler or interpreter. Its function is to perform grammar checking, analysis, and build a data structure (such as an abstract syntax tree AST) consisting of input words. ANTLR (Another Tool for Language Recognition) is a common open source grammar parser in the Java field. Developers can build their own domain language based on ANTLR, thereby helping business personnel without a R&D background to build business processing logic more flexibly and conveniently.

[0004] In the "Data Cleaning Framework Based on Dynamically Configurable Rules in the Context of Big Data", developers proposed a data processing framework that combines dynamically configurable rules with data cleaning. This technical solution divides rule types into three types: Drools rules, regular expression rules, and function rules, and implements the corresponding rule engine to parse the logical expression of the rule, and then calls the interfaces of the three rules based on the rule type to perform data processing.

[0005] This technical solution has the following deficiencies:

[0006] 1. In this technical solution, there is a lack of flexibility in configuring data processing rules because the rule engine implemented by this invention supports fewer grammatical rules;

[0007] 2. In this technical solution, the data processing rules’ effectiveness mechanism is not timely and convenient enough. After modifying the data processing rules, the program needs to be restarted because the solution uses configuration files to arrange and call the data processing rules.

[0008] 3. In this technical solution, the writing of data processing rules is not simple enough, because in addition to configuring the data matching calculation rules, it is also necessary to configure the data processing method (such as modification or discarding).

[0009] In the existing big data processing technology solutions based on configurable rules, the modification of runtime data processing logic lacks flexibility and convenience. For example: 1. The flexibility of data processing rule configuration is insufficient, and fewer grammatical rules are supported; 2. The data processing rules are not effective in a timely and convenient manner. For example, after modifying the data processing rules, the deployment code needs to be recompiled or the program needs to be restarted; 3. The writing of data processing rules is not simple enough. Summary of the invention

[0010] In order to overcome the shortcomings of the prior art, the present invention provides a data processing architecture method based on ANTLR 4, which supports richer rule grammars and provides a variety of preset data calculation rules. At the same time, it incorporates a hot loading mechanism for periodically updating data processing rules, and can provide a flexible configuration, compilation-free and hot loading technical solution for the real-time and offline big data processing logic arrangement. After temporarily modifying the data processing logic during the big data processing process, there is no need to recompile and deploy or restart the running program to make the rules effective in a timely and convenient manner and obtain the processing results of the latest data processing logic.

[0011] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0012] Step 1: Syntax rule definition and configuration;

[0013] Define grammar and keywords according to domain business requirements, then create grammar rule file XLogic.g and keyword file XLogic.tokens, and generate grammar parsing related tool classes through ANTLR;

[0014] Step 2: Write business expressions for data processing;

[0015] Write data processing expressions based on the grammar rule file in step 1, and configure the written rules for the data to be processed;

[0016] Step 3: Build the expression input stream;

[0017] Periodically read the data processing expressions written in step 2 from the database, and create an ANTLRInputStream stream for the changed expressions;

[0018] Step 4: Lexical analysis;

[0019] Process the ANTLRInputStream stream, identify and segment the keywords in the expression, and then create the lexical analysis result CommonTokenStream stream based on the recognition results;

[0020] Step 5: Syntax analysis;

[0021] Process the CommonTokenStream stream of the lexical analysis result in step 4, and then build a syntax tree based on the syntax analysis result;

[0022] Step 6: Build the expression tree;

[0023] Traverse the syntax tree generated in step 5 and construct an expression tree XNode that can perform data processing;

[0024] Step 7: Data structure conversion;

[0025] Use the spark stream API to read real-time or offline big data from Kafka and convert the data into a KMessage structure;

[0026] Step 8: Create a data processing context;

[0027] Use the data of the KMessage structure constructed in step 7 and the preset data processing function to build a data processing context;

[0028] Step 9: Data processing;

[0029] The expression tree XNode constructed in step 6 is traversed by a traversal algorithm, and the data processing context context constructed in step 8 is processed to finally obtain the data processing result;

[0030] Step 10: Data output;

[0031] Convert the processed data into a byte stream and then output the final result to Kafka.

[0032] Preferably, the periodic reading of the data processing expression written in step 2 from the database in step 3 is completed by an expression configuration and parsing module.

[0033] Preferably, the processing of the ANTLRInputStream stream in step 4 is performed by a lexical analyzer XLogicLexer generated based on custom rules.

[0034] Preferably, in step 5, the processing of the lexical analysis result CommonTokenStream in step 4 is performed by a syntax analyzer XLogicParser generated based on custom rules.

[0035] Preferably, in step 6, traversing the syntax tree generated in step 5 is performed using a custom syntax tree traverser XLogicVisitorImpl through a visitor pattern.

[0036] The beneficial effects of the present invention are as follows:

[0037] The present invention overcomes the shortcomings of the traditional big data processing business, that is, the configuration of the data processing logic is not flexible enough, and the data processing logic is difficult to be effective in a timely and convenient manner after modification, and provides richer grammatical rules. Users can build data processing rules based on the customized dynamic domain language and the business needs of the current data processing. The present invention also adopts a hot loading mechanism for periodically loading data processing rules to call the constructed data processing rules without recompiling the deployment program or restarting the program, thereby greatly improving the timeliness and convenience of the big data processing rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] The purpose of the present invention is to provide a runtime data processing architecture method based on ANTLR 4, which supports richer rule grammars and provides a variety of preset data calculation rules, and integrates a hot loading mechanism for periodically updating data processing rules, which can provide a flexible configuration, compilation-free and hot loading technical solution for real-time and offline big data processing logic arrangement. After temporarily modifying the data processing logic during the big data processing process, it is not necessary to recompile and deploy or restart the running program to make the rules effective in a timely and convenient manner and obtain the processing results of the latest data processing logic.

[0041] Step 101: Syntax rule definition and configuration;

[0042] Define grammar and keywords according to domain business requirements, then create grammar rule file XLogic.g and keyword file XLogic.tokens, and generate grammar parsing related tool classes through ANTLR;

[0043] Step 102: Write a business expression for data processing;

[0044] Write a data processing expression based on the self-defined grammar rules in step 101, and configure the written rules for the data to be processed. For example, the expression "(!(isBlank($key1))AND($key2 like"[0-9]+"))?value1:value2" is used to determine whether the value of "key1" is not blank and "key2" is a pure Arabic numeral. If yes, return (value1), otherwise return "value2";

[0045] Step 103: construct an expression input stream;

[0046] like Figure 1 As shown, the expression configuration and parsing module first periodically reads the data processing expression written in step 102 from the database, and creates an ANTLRInputStream stream for the changed expression;

[0047] Step 104: lexical analysis;

[0048] Use the lexical analyzer XLogicLexer generated based on custom rules to process the ANTLRInputStream stream, identify and segment the keywords in the expression, and then create the CommonTokenStream stream based on the recognition results;

[0049] Step 105: Syntax analysis;

[0050] Use the syntax analyzer XLogicParser generated based on the custom rules to process the CommonTokenStream stream of the lexical analysis result in step 104, and then build a syntax tree based on the syntax analysis result;

[0051] Step 106: construct an expression tree;

[0052] Use a custom syntax tree traverser XLogicVisitorImpl to traverse the syntax tree generated in step 105 through a visitor pattern and construct an expression tree XNode that can perform data processing;

[0053] Step 107: data structure conversion;

[0054] Use the spark stream API to read real-time or offline big data from Kafka and convert the data into a KMessage structure;

[0055] Step 108: Create a data processing context;

[0056] Use the KMessage structure data constructed in step 107 and the preset data processing function to construct a data processing context context;

[0057] Step 109: Data processing;

[0058] This step uses a subsequent traversal algorithm to traverse the expression tree XNode constructed in step 106 and processes the data processing context context constructed in step 108 to finally obtain the data processing result;

[0059] Step 110: Data output;

[0060] Convert the processed data into a byte stream and then output the final result to Kafka.

[0061] The key point of the present invention is to realize a dynamic domain language based on ANTLR 4, and combine the dynamic domain language with big data processing rule arrangement, thereby providing a data processing architecture method based on ANTLR 4.

[0062] The invention is intended to protect:

[0063] 1. Domain dynamic language based on ANTLR 4, including relevant grammar rules, preset data processing rules, and methods for parsing expressions and building expression trees based on custom rules;

[0064] 2. The method flow of runtime orchestration and hot loading of big data processing rules based on custom domain dynamic language.

[0065] In the existing big data processing scheme, the modification of data processing logic lacks flexibility, and the loading mechanism of data processing rules is not timely and convenient enough. However, in the big data processing business, the data processing logic needs to be flexibly adjusted frequently to achieve the ideal processing effect, and the rules also need to be effective in time, so the flexibility of data processing rule configuration and the convenience and timeliness of rule loading and effectiveness are greatly challenged. The present invention implements a domain dynamic language based on ANTLR 4, and constructs a runtime rule arrangement and dynamic effectiveness mechanism for big data processing based on the language, so that the arrangement of big data processing rules is more flexible and convenient, and the effectiveness is more timely and convenient.

Claims

1. A data processing architecture method based on ANTLR 4, characterized in that: The steps include: Step 1: Syntax rule definition and configuration; Define grammar and keywords according to domain business requirements, then create grammar rule file XLogic.g and keyword file XLogic.tokens, and generate grammar parsing related tool classes through ANTLR; Step 2: Write business expressions for data processing; Write data processing expressions based on the grammar rule file in step 1, and configure the written rules for the data to be processed; Step 3: Build the expression input stream; Periodically read the data processing expressions written in step 2 from the database, and create an ANTLRInputStream stream for the changed expressions; Step 4: Lexical analysis; Process the ANTLRInputStream stream, identify and segment the keywords in the expression, and then create the lexical analysis result CommonTokenStream stream based on the recognition results; Step 5: Syntax analysis; Process the CommonTokenStream stream of the lexical analysis result in step 4, and then build a syntax tree based on the syntax analysis result; Step 6: Build the expression tree; Traverse the syntax tree generated in step 5 and construct an expression tree XNode that can perform data processing; Step 7: Data structure conversion; Use the spark stream API to read real-time or offline big data from Kafka and convert the data into a KMessage structure; Step 8: Create a data processing context; Use the data of the KMessage structure constructed in step 7 and the preset data processing function to build a data processing context; Step 9: Data processing; The expression tree XNode constructed in step 6 is traversed by a traversal algorithm, and the data processing context context constructed in step 8 is processed to finally obtain the data processing result; Step 10: Data output; Convert the processed data into a byte stream and then output the final result to Kafka.

2. A data processing architecture method based on ANTLR 4 according to claim 1, characterized in that, The step 3 of periodically reading the data processing expression written in step 2 from the database is completed through the expression configuration and parsing module.

3. A data processing architecture method based on ANTLR 4 according to claim 2, characterized in that, The processing of the ANTLRInputStream stream in step 4 is performed by a lexical analyzer XLogicLexer generated based on custom rules.

4. A data processing architecture method based on ANTLR 4 according to claim 3, characterized in that, The processing of the lexical analysis result CommonTokenStream in step 4 in step 5 is performed by a syntax analyzer XLogicParser generated based on custom rules.

5. A data processing architecture method based on ANTLR 4 according to claim 4, characterized in that, In step 6, the syntax tree generated in step 5 is traversed by using a custom syntax tree traverser XLogicVisitorImpl through the visitor pattern.

Images