Method and device for processing function expression heterogeneous data source

Abstract

The invention relates to the technical field of data processing, and discloses a function expression heterogeneous data source processing method and device.The method comprises the steps that a first query statement and a data processing expression in a data processing text are obtained, and the first query statement is set according to the same self-defined grammar rule; converting the first query statement into a corresponding abstract syntax tree according to a preset analysis rule, traversing nodes of the abstract syntax tree to generate a second query statement, querying the corresponding databases based on the second query statement, obtaining target data of a plurality of heterogeneous databases, and performing integration calculation on the target data by using an expression. According to the method, the first query statement only has one rule, after the first query statement is converted into the second query statement, the second query statement can be matched with the respective database, and research personnel only need to learn the grammar rule of the first query statement to simplify the process of manual data calculation; and the data processing efficiency during large-scale data integration based on the heterogeneous data source is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for processing heterogeneous data sources of function expressions. Background Art

[0002] Currently, in the big data log scenario, in order to adapt to different business scenarios, data is stored in different heterogeneous data sources, including relational databases and non-relational databases, and the databases contain structured and unstructured data, hot and cold data, etc. Each corresponds to a different heterogeneous database, and in the analysis and statistics scenario, the data in these heterogeneous data sources needs to be jointly calculated.

[0003] In the existing technology, for this type of data joint processing scenario, users need to be familiar with the actual grammatical query rules corresponding to multiple databases. However, the grammatical query rules of different databases are different. When actually statistically analyzing heterogeneous data, it is necessary to perform large, complex and repetitive data processing, which is time-consuming and labor-intensive and may lead to errors, reducing the efficiency of large-scale data integration. Summary of the invention

[0004] In view of this, the present invention provides a method and device for processing heterogeneous data sources of function expressions to solve the problem of how to simplify the processing operations of heterogeneous data sources of function expressions while ensuring data accuracy, thereby improving data processing efficiency when performing large-scale data integration based on heterogeneous data sources.

[0005] In a first aspect, the present invention provides a method for processing heterogeneous data sources of function expressions, the method comprising: obtaining a data query function processing text, wherein the data query function processing text comprises first query statements and data processing expressions corresponding to a plurality of heterogeneous databases respectively; the first query statements corresponding to the plurality of heterogeneous databases respectively are set based on a preset same custom grammar rule;

[0006] According to the preset parsing rule corresponding to the preset same custom grammar rule, convert each of the first query statements into a corresponding abstract syntax tree;

[0007] Traversing all nodes of each abstract syntax tree to generate a second query statement for a heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases;

[0008] Based on the plurality of second query statements, the corresponding heterogeneous databases are respectively queried to obtain target data from the plurality of heterogeneous databases;

[0009] The target data is integrated and calculated using the data processing expression to obtain a data integration result of a heterogeneous database.

[0010] In the embodiment of the present invention, the first query statement in the data query function processing text can realize the query operation on multiple heterogeneous databases through unified custom grammar rules, wherein the custom grammar rules can solve the grammatical differences between different databases. Regardless of the database, the same grammar rules can be used to express the user's own query requirements. By converting the first query statement into an abstract syntax tree, traversing the abstract syntax tree nodes to generate a second query statement that conforms to the actual grammar rules of each heterogeneous database, the query intent of the first query statement can be restored, replacing the process of manually writing each database statement, and integrating and calculating the query results from different databases through a unified data processing expression. Since there is a data processing expression in the first step of the data query function processing text, the process of manual data calculation can be further simplified. The embodiment of the present invention simplifies the processing operation of the function expression heterogeneous data source while ensuring data accuracy, thereby improving the data processing efficiency when performing large-scale data integration based on heterogeneous data sources.

[0011] In an optional implementation, the preset parsing rules include: lexical rules and grammatical combination rules;

[0012] The converting each of the first query statements into a corresponding abstract syntax tree according to the preset parsing rule corresponding to the preset same custom grammar rule specifically includes:

[0013] Performing lexical analysis on the first query statement according to the lexical rules in the preset parsing rules to obtain a lexical result;

[0014] Performing grammatical combination analysis on the first query statement according to the grammatical combination rule in the preset parsing rule to obtain a grammatical combination result;

[0015] The grammar combination results are used as tree branches, and the lexical results are used as tree nodes to construct an abstract syntax tree containing multiple nodes.

[0016] The embodiment of the present invention sets lexical rules and grammatical combination rules as preset parsing rules and performs lexical analysis and grammatical combination analysis on the first query statement based on the two rules, thereby constructing an abstract syntax tree including lexical results and grammatical combination results, so that the first query statement can be accurately parsed into a structured tree representation, and the query statement with unified grammatical rules is converted into an intermediate representation, which in turn provides a structural basis for the subsequent generation of actual query statements for different databases.

[0017] In an optional implementation, traversing all nodes of each abstract syntax tree to generate a second query statement of a heterogeneous database corresponding to each abstract syntax tree specifically includes:

[0018] Perform a depth-first traversal of the tree branches of the current abstract syntax tree, and obtain the semantic unit in the lexical result of the current tree node each time a tree node is traversed;

[0019] After obtaining the semantic units in all tree nodes of the current abstract syntax tree, multiple semantic units are assembled into a second query statement corresponding to the current abstract syntax tree according to the actual syntax rules of the heterogeneous database corresponding to the current abstract syntax tree, wherein one abstract syntax tree corresponds to one second query statement.

[0020] The embodiment of the present invention obtains semantic units by performing a depth-first traversal on the abstract syntax tree, and assembles these semantic units according to the actual syntax rules of the target database, thereby converting the abstract syntax tree into a query statement that conforms to the syntax of a specific database, which is equivalent to restoring the intermediate representation to the actual database query statement, thereby realizing automatic and rapid conversion from a unified custom syntax to multiple other database syntaxes, and avoiding the work of manually writing multiple query statements.

[0021] In an optional implementation, the data query function processing text also includes link data information corresponding to different heterogeneous databases;

[0022] Before respectively querying the corresponding heterogeneous databases based on the plurality of second query statements to obtain the target data from the plurality of heterogeneous databases, the method further includes:

[0023] A heterogeneous database link pool is established according to the link data information, and the database link pool contains different link paths with different heterogeneous databases.

[0024] The embodiment of the present invention includes link data information in the data query function processing text and establishes a heterogeneous database link pool, thereby achieving unified management of multiple heterogeneous database connections, uniformly maintaining multiple database connections, and providing path support for subsequent parallel and fast query of multiple databases.

[0025] In an optional implementation, querying the corresponding heterogeneous databases based on the plurality of second query statements respectively to obtain the target data from the plurality of heterogeneous databases specifically includes:

[0026] Based on different link paths in the heterogeneous database link pool, different second query statements are distributed to corresponding heterogeneous databases, and target data returned by each heterogeneous database is received.

[0027] The embodiment of the present invention distributes query statements and receives return data through different link paths in a heterogeneous database link pool, thereby realizing parallel query operations on multiple heterogeneous databases, efficiently acquiring a large amount of data from multiple heterogeneous databases, and further improving the overall efficiency of heterogeneous database queries.

[0028] In an optional implementation, the data processing expression is composed of an operator and a heterogeneous data identification variable, wherein one heterogeneous data identification variable corresponds to target data of one heterogeneous database;

[0029] The integrating calculation of the target data by using the data processing expression to obtain the data integration result of the heterogeneous database specifically includes:

[0030] The target data returned by each heterogeneous database is substituted into the corresponding heterogeneous data identification variable, and the data integration result of the heterogeneous database is calculated through operators.

[0031] The embodiment of the present invention substitutes the target data obtained by the query into the calculation through a data processing expression composed of operators and heterogeneous data identification variables, automatically integrates the query results of multiple heterogeneous databases, simplifies the heterogeneous data integration calculation process, and thus improves the automation level of large-scale heterogeneous database data integration processing and improves data integration efficiency.

[0032] In a second aspect, the present invention provides a device for processing heterogeneous data sources of function expressions, the device comprising:

[0033] A query text acquisition module is used to acquire a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to a plurality of heterogeneous databases respectively; the first query statements corresponding to the plurality of heterogeneous databases are set based on a preset same custom grammar rule;

[0034] A syntax tree conversion module, configured to convert each of the first query statements into a corresponding abstract syntax tree according to a preset parsing rule corresponding to the preset same custom syntax rule;

[0035] A query statement generation module, used to traverse all nodes of each abstract syntax tree and generate a second query statement for a heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases;

[0036] A data query execution module, configured to query the corresponding heterogeneous databases based on the plurality of second query statements respectively, and obtain target data from the plurality of heterogeneous databases;

[0037] The data integration processing module is used to use the data processing expression to perform integration calculation on the target data to obtain the data integration result of the heterogeneous database.

[0038] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method for processing heterogeneous data sources of function expressions of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for processing heterogeneous data sources of function expressions of the first aspect or any corresponding embodiment thereof.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for processing heterogeneous data sources of function expressions of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a first flow chart of a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0043] Figure 2 is a second flow chart of a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0044] Figure 3 It is a schematic diagram of an abstract syntax tree structure of a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0045] Figure 4 is a third flow chart of the method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0046] Figure 5 It is a schematic diagram of the implementation principle of the method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0047] Figure 6is a schematic diagram of a database converter kernel according to a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0048] Figure 7 is a structural block diagram of a device for processing heterogeneous data sources of function expressions according to an embodiment of the present invention;

[0049] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] In the scenario of big data log analysis, Redis and ClickHouse are two common databases, which are suitable for different scenario requirements. Redis is an in-memory database used in scenarios with high real-time requirements. ClickHouse is a column storage database used to process analytical queries on large-scale data, and scenarios that require fast aggregation and statistics.

[0052] In the scenario of heterogeneous data sources, in order to jointly calculate the data in Redis and ClickHouse, R&D personnel need to write query statements for the two databases respectively. For Redis, the query statements are such as SET key value, GET key, etc., where key represents the key in the Redis data storage structure and value represents the value; for ClickHouse, the query statements are such as SELECT field 1, COUNT (*) as field 2 FROM table GROUP BY field 1. It can be clearly seen that the query statements of the two databases are different. For the data integration process, it is not enough to just obtain data. It is still necessary to write the overall processing logic for the two data later. If more types of input data are expanded, not only the database query language needs to be learned, but also the overall processing logic needs to be changed, which is time-consuming and laborious.

[0053] According to an embodiment of the present invention, an embodiment of a method for processing heterogeneous data sources of function expressions is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] In this embodiment, a method for processing heterogeneous data sources of function expressions is provided, which can be used in the above-mentioned computer. Figure 1 is a flowchart of a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0055] Step S101, obtaining a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to multiple heterogeneous databases respectively; the first query statements corresponding to the multiple heterogeneous databases are set based on the same preset custom grammar rules.

[0056] It should be noted that the execution subject of the embodiment of the present invention is the execution engine, wherein the data query function processing text refers to the text information storing the user's query requirements, including the statement for querying the data source and the subsequent calculation logic to be executed, the heterogeneous database refers to the database of different types and different syntaxes, such as Redis, ClickHouse, etc., the first query statement refers to the query statement written by the user based on the custom grammar rules, which is used to describe the query requirements for different databases, the preset same custom grammar rules refers to a set of unified grammar rules defined for all databases, and the data processing expression refers to the calculation logic that the user needs to perform after querying the data, such as the simplest addition, subtraction, multiplication, division and other operations.

[0057] The following is a further explanation of the above content with examples:

[0058] Example 1, the function processes the text as follows:

[0059] A=ct_int(field_get("payout")):"DS-1";

[0060] B=ct_float(ext_regex("income","(.*)(.*)","1")):"DS-2";

[0061] C = 100;

[0062] A*B+C*A+B / C;

[0063] Among them, ct_int(field_get("payout")):"DS-1" means to query the field payout from the data source DS-1 and convert it into an integer. ct_float(ext_regex("income","(.*)(.*)",

[0064] "1")):"DS-2" means querying the field income from the data source DS-2, extracting the first matching value through regular expression and converting it into a floating point number. The calculation expression A*B+C*A+B / C represents the integrated calculation logic for the query results.

[0065] Example 2: DS-1 can represent the unique identifier of the Redis database, and DS-2 represents the unique identifier of ClickHouse.

[0066] Example 3: ct_int(field_get("payout")):"DS-1" is the first query statement.

[0067] Example 4: ct_int(XX): "XX" indicates a preset custom syntax rule, which means converting the database query data into integer data. It is expressed in the form of a function in the application, where the content of XX can be replaced according to different databases and different query operations.

[0068] Step S102: convert each first query statement into a corresponding abstract syntax tree according to a preset parsing rule corresponding to the same preset custom grammar rule.

[0069] It should be noted that the preset parsing rules corresponding to the same custom grammar rule refer to a set of parsing methods based on a defined unified grammar rule, such as Antlr4 parsing grammar rules. Each first query statement corresponds to a query statement written by a user for a database, and each statement corresponds to a target database. The abstract syntax tree (AST) refers to the tree structure generated after the first query statement is parsed, and each node on the tree represents a component of the statement.

[0070] Specifically, the parsing process can be completed by Antlr4 or PLY database query statement syntax analysis tools. Antlr4 is a lexical and grammatical parser code generator. Antlr4 provides a relatively simple and powerful grammar file debugging function, AST tree traversal method, and multi-language support; PLY is a pure Python implementation of the lexical analyzer and grammatical analyzer generator, the principle can be based on the Lex and Yacc tools. It is suitable for Python development scenarios and is used to build parsers, compilers, interpreters, and other language processing tools.

[0071] Exemplarily, the following query statement ct_int(field_get("payout")):"DS-1" is parsed into an abstract syntax tree, whose tree nodes may include: root node ct_int, child node 1: field_get, child node 2: "payout", child node 3: target data source "DS-1".

[0072] It is understandable that through the analysis of AST, the structure of the query statement can be clearly represented, which is convenient for subsequent translation operations.

[0073] Step S103, traversing all nodes of each abstract syntax tree, generating a second query statement of the heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases.

[0074] It should be noted that the second query statement refers to a query statement generated based on the actual grammatical rules of each heterogeneous database. Different from the first query statement, the second query statement is a statement that can be directly executed by the corresponding database.

[0075] Specifically, according to the rules of different databases, the abstract syntax tree is translated into the corresponding query statement using the syntax converter of each database (such as Redis converter, ClickHouse converter).

[0076] For example, the first query statement is ct_int(field_get("payout")):"DS-1", the target database is Redis, and the second query statement generated is GET payout. It is understandable that different databases have different grammatical rules. Through the traversal of the abstract syntax tree and the logic of the converter, query statements adapted to the corresponding database can be automatically generated to avoid manual writing by users.

[0077] Step S104: query the corresponding heterogeneous databases based on the multiple second query statements respectively to obtain target data from the multiple heterogeneous databases.

[0078] It should be noted that the target data from multiple heterogeneous databases refers to the result data returned after each database executes the query statement.

[0079] Specifically, the actual query process is completed through different data source adapters. The adapters of different databases are responsible for establishing connections, sending statements, and receiving data. For example, the Redis query statement GET payout returns the target data. It is understandable that the adapter solves the differences in the connection or data access process of different databases. Users only need to pay attention to the query results, without worrying about the specific query process.

[0080] Step S105, using data processing expressions to perform integration calculations on the target data to obtain data integration results of the heterogeneous databases.

[0081] It should be noted that the integration result refers to the result of calculating the data returned by different databases, such as simple data processing expressions such as sum, average, product, etc., and may also include formulas of complex calculation symbols.

[0082] It is understandable that by parsing user-defined expressions and combining them with the queried target data, complex cross-database calculation logic is completed, which avoids the complexity of manual integration calculations by users and improves data processing efficiency.

[0083] In the embodiment of the present invention, the first query statement in the data query function processing text can realize the query operation on multiple heterogeneous databases through unified custom grammar rules, wherein the custom grammar rules can solve the grammatical differences between different databases. Regardless of the database, the same grammar rules can be used to express the user's own query requirements. By converting the first query statement into an abstract syntax tree, traversing the abstract syntax tree nodes to generate a second query statement that conforms to the actual grammar rules of each heterogeneous database, the query intent of the first query statement can be restored, replacing the process of manually writing each database statement, and integrating and calculating the query results from different databases through a unified data processing expression. Since there is a data processing expression in the first step of the data query function processing text, the process of manual data calculation can be further simplified. The embodiment of the present invention simplifies the processing operation of the function expression heterogeneous data source while ensuring data accuracy, thereby improving the data processing efficiency when performing large-scale data integration based on heterogeneous data sources.

[0084] In this embodiment, a method for processing heterogeneous data sources of function expressions is provided, which can be used in the above-mentioned computer. Figure 2 is a flowchart of a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0085] Step S201, obtaining a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to multiple heterogeneous databases respectively; the first query statements corresponding to multiple heterogeneous databases are set based on the same preset custom grammar rules. For details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0086] Step S202: convert each first query statement into a corresponding abstract syntax tree according to a preset parsing rule corresponding to the same preset custom grammar rule.

[0087] Among them, the preset parsing rules include: lexical rules and grammatical combination rules;

[0088] Specifically, the above step S202 includes:

[0089] Step S2021, performing lexical analysis on the first query statement according to the lexical rules in the preset parsing rules to obtain a lexical result;

[0090] Step S2022, performing grammatical combination analysis on the first query statement according to the grammatical combination rule in the preset parsing rule to obtain a grammatical combination result;

[0091] Step S2023, using the grammatical combination results as tree branches and the lexical results as tree nodes, to construct an abstract syntax tree containing multiple nodes.

[0092] It should be noted that lexical rules refer to the smallest semantic unit (Token) used to define custom grammar rules. For example, function names such as ct_int and field_get can be defined as identifiers. Grammar combination rules refer to the rules of dependency relationships such as inclusion and parallelism of each smallest semantic unit. In practical applications, G4 grammar can be used for parsing. The specific process is as follows: Token segmentation is automatically completed according to the set G4 grammar rules. The output of lexical analysis is a Token sequence, which represents the components and types of the input query statement; the components and types in the Token sequence obtained by lexical analysis are parsed level by level, and finally a structured representation (grammar combination result) that conforms to the grammar rules is generated. The structured representation contains the grammatical hierarchy of the statement (such as parallel hierarchy and nested hierarchy, etc.).

[0093] In this embodiment, the example code of the G4 grammar rule text content set based on the second query statement is as follows:

[0094]

[0095] :'false';

[0096] / / Numerical value lexical definition '_'? ([0-9]*'.')? [0-9]+; NUMBER1 / --String text definition: ('“'|'\')~[\r\n']*('"'|'\”); TEXT.

[0097] The embodiment of the present invention sets lexical rules and grammatical combination rules as preset parsing rules and performs lexical analysis and grammatical combination analysis on the first query statement based on the two rules, thereby constructing an abstract syntax tree including lexical results and grammatical combination results, so that the first query statement can be accurately parsed into a structured tree representation, and the query statement with unified grammatical rules is converted into an intermediate representation, which in turn provides a structural basis for the subsequent generation of actual query statements for different databases.

[0098] Step S203, traversing all nodes of each abstract syntax tree, generating a second query statement of the heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases.

[0099] Specifically, the above step S203 includes:

[0100] Step S2031, performing a depth-first traversal according to the tree branches of the current abstract syntax tree, and obtaining a semantic unit in the lexical result of the current tree node each time a tree node is traversed;

[0101] Step S2032, after obtaining the semantic units in all tree nodes of the current abstract syntax tree, assemble multiple semantic units into a second query statement corresponding to the current abstract syntax tree according to the actual syntax rules of the heterogeneous database corresponding to the current abstract syntax tree, wherein one abstract syntax tree corresponds to one second query statement.

[0102] Specifically, step S2031 includes: (1) starting from the root node of the abstract syntax tree, traversing each node in a depth-first order; (2) extracting semantic unit information from each traversed node, including: function name, parameter value, data source identifier, operator symbol, and semantic words other than any of the above items.

[0103] Step S2032 includes: (1) determining the basic structure and grammatical format of the query statement according to the grammatical rules of the target database; and (2) assembling the semantic units obtained in step S2031 according to the grammatical rules of the target database.

[0104] The assembly process is as follows: convert the function name into the corresponding database operation command; adjust the position and format of the parameters, and generate a second query statement that conforms to the syntax rules of the target database according to the position and format of the database operation command and the parameters. For example, taking a statement str_join(”,str_lowercase('aBc5')'d'1,true) as an example, the generated abstract syntax tree is as follows Figure 3 As shown, expression:1 is the root node corresponding to the next methodExec node. The next layer of the methodExec node includes two nodes, methodName and methodExecArguments. The next node of methodName is ID:str join. The next layer of methodExecArguments includes five nodes, namely expression:2, expression:1, expression:2, expression:3, and expression:4.

[0105] Among them, the two expression:2 nodes correspond to the next layer of TEXT:',' ​​node and the next layer of TEXT:'d' node respectively, the expression:3 node corresponds to the next layer of NUMBER:1 node, the expression:4 node corresponds to the next layer of BOOL:true node, the expression:1 node corresponds to the next layer of methodExec node, the next layer of methodExec node includes two nodes methodName and methodName, methodName corresponds to ID:str lowercase node and expression:2 node, and the next layer of expression:2 node is TEXT:'aBc5' node.

[0106] Similarly, whether it is a database query statement or other non-query statement, based on the information on the tree nodes and the grammar combination rules of each database, the semantic units in the nodes are combined to obtain a second query statement. It can be understood that by traversing and assembling, the query statements under the unified grammar rules can be converted into query statements that can be actually executed by each database.

[0107] Step S204: Based on the multiple second query statements, the corresponding heterogeneous databases are queried respectively to obtain the target data from the multiple heterogeneous databases. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0108] Step S205: Use data processing expressions to perform integration calculations on the target data to obtain data integration results for heterogeneous databases. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0109] The embodiment of the present invention obtains semantic units by performing a depth-first traversal on the abstract syntax tree, and assembles these semantic units according to the actual syntax rules of the target database, thereby converting the abstract syntax tree into a query statement that conforms to the syntax of a specific database, which is equivalent to restoring the intermediate representation to the actual database query statement, thereby realizing automatic and rapid conversion from a unified custom syntax to multiple other database syntaxes, and avoiding the work of manually writing multiple query statements.

[0110] This embodiment provides a method for processing heterogeneous data sources of function expressions, which can be used in the above-mentioned computer. Figure 4 is a flowchart of a method for processing heterogeneous data sources of function expressions according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0111] Step S401, obtaining a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to multiple heterogeneous databases; the first query statements corresponding to multiple heterogeneous databases are set based on the same preset custom grammar rules. For details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0112] Step S402: Convert each first query statement into a corresponding abstract syntax tree according to a preset parsing rule corresponding to the same preset custom syntax rule. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0113] Step S403, traversing all nodes of each abstract syntax tree, generating a second query statement for the heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases.

[0114] Step S404: establishing a heterogeneous database link pool according to the link data information, wherein the database link pool includes different link paths to different heterogeneous databases.

[0115] In this embodiment, the data query function processing text also includes the link data information corresponding to different heterogeneous databases. Examples of link data information of two databases Ds-1 and DS-2 are as follows:

[0116] {"name':"Ds-1""type":"REDIS""password":"xxxx""username:"xxxxx""url':"127.0.0.1:6379"database":0

[0117] };

[0118] {"name:"DS-2","type":"CLICKHOUSE""password":"xxoxx""username:"xxxx"."url':"127.0.0.1:9090"database":"default""table":'t test1"

[0119] };

[0120] Specifically, a link path of the database is established through the type, url, database (optionally username and password) in the above link data information. The path creation process includes: parsing the link data information to extract the type, url, database, username and password; selecting a corresponding database connection driver according to the database type, using the type, url, database, username and password to create a connection to the corresponding database through the driver, storing the created connection in the database connection pool, and reusing the path when connecting to the database again later.

[0121] The embodiment of the present invention includes link data information in the data query function processing text and establishes a heterogeneous database link pool, thereby achieving unified management of multiple heterogeneous database connections, uniformly maintaining multiple database connections, and providing path support for subsequent parallel and fast query of multiple databases.

[0122] Step S405 , querying the corresponding heterogeneous databases based on the multiple second query statements respectively, and obtaining target data from the multiple heterogeneous databases.

[0123] Specifically, step S405 includes: based on different link paths in the heterogeneous database link pool, distributing different second query statements to corresponding heterogeneous databases, and receiving target data returned by each heterogeneous database.

[0124] Exemplarily, taking the link path in step S404 as an example, the process of distributing different second query statements to corresponding heterogeneous databases includes: obtaining a database connection that matches the data source identifier (Ds-1 or DS-2) corresponding to each second query statement from a database connection pool; sending the second query statement to the corresponding heterogeneous database through the obtained database connection; waiting for and receiving result data returned by each database after executing the query.

[0125] The embodiment of the present invention distributes query statements and receives return data through different link paths in a heterogeneous database link pool, thereby realizing parallel query operations on multiple heterogeneous databases, efficiently acquiring a large amount of data from multiple heterogeneous databases, and further improving the overall efficiency of heterogeneous database queries.

[0126] Step S406, using data processing expressions to perform integration calculations on the target data to obtain data integration results of the heterogeneous databases.

[0127] The data processing expression is composed of operators and heterogeneous data identification variables, wherein one heterogeneous data identification variable corresponds to target data of a heterogeneous database;

[0128] Specifically, step S406 includes: substituting the target data returned by each heterogeneous database into the corresponding heterogeneous data identification variable, and calculating the data integration result of the heterogeneous database through the operator. Exemplary, the data processing expression is: A*B+C*A+B / C; wherein A, B, and C all represent heterogeneous data identification variables, and C is a preset fixed value of 100. If the Redis database returns data 100 and the ClickHouse database returns data 200, then the data integration result of the heterogeneous database calculated based on the above expression is: 30002. The data processing expression of the present invention is only used as an example, and is not limited to this expression in actual application.

[0129] The embodiment of the present invention substitutes the target data obtained by the query into the calculation through a data processing expression composed of operators and heterogeneous data identification variables, automatically integrates the query results of multiple heterogeneous databases, simplifies the heterogeneous data integration calculation process, and thus improves the automation level of large-scale heterogeneous database data integration processing and improves data integration efficiency.

[0130] The implementation ideas and principles of the embodiments of the present invention in the specific execution engine process can be referred to Figure 5 ,The processing flow of the execution engine is as follows: after the function expression text is input into the execution engine, it is received and processed by the processing and ingestion function expression converter; the XXVisitor component in the converter traverses the AST syntax tree generated by Antlr4 to convert the expression syntax into processing calculation scripts of different data sources; the aggregation calculator uses Antlr4 to parse the calculation expression and save it to the calculation cache queue; the data source data ingestion adapter sends the converted processing ingestion script to the corresponding data source for calculation through the heterogeneous data source connection pool; after the calculation of each data source is completed, the result is returned to the expression calculator of the aggregation calculation. After the calculation cache variable rendering and the expression execution calculation, the execution engine outputs the final result.

[0131] It should be noted that, please refer to Figure 6 In the embodiment of the present invention, a dedicated converter XX_Converter is configured for each heterogeneous data source, such as Redis_Converter, etc. Its core logic is to use Antlr4 to convert the input function expression into a processing calculation script of the target data source. Specifically, after the function expression text is input into XX_Converter, the AST syntax tree is generated by the lexical and grammar generator of Antlr4, and then the AST tree is converted into the calculation script of the corresponding data source through the corresponding visitor component (XX_Visitor, such as Redis_Visitor, etc.). XX_Converter refers to a dedicated converter for different data sources, and XX_Visitor refers to the visitor component for different data sources.

[0132] Through the unified function expression syntax specification, the embodiment of the present invention significantly reduces the learning cost of processing and computing languages ​​for multiple heterogeneous data sources. By dividing the complex calculations between heterogeneous data sources into two clear levels, expression conversion and data source calculation, it can effectively reduce the redundancy of heterogeneous data sources, reduce maintenance costs, and simplify the complexity of data integration modules in practical applications.

[0133] In this embodiment, a processing device for heterogeneous data sources of function expressions is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0134] This embodiment provides a processing device for heterogeneous data sources of function expressions, such as Figure 7 As shown, including:

[0135] The query text acquisition module 701 is used to acquire a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to a plurality of heterogeneous databases respectively; the first query statements corresponding to the plurality of heterogeneous databases are set based on a preset same custom grammar rule;

[0136] A syntax tree conversion module 702, configured to convert each first query statement into a corresponding abstract syntax tree according to a preset parsing rule corresponding to the same preset custom syntax rule;

[0137] A query statement generating module 703 is used to traverse all nodes of each abstract syntax tree and generate a second query statement for a heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases;

[0138] A data query execution module 704 is used to query the corresponding heterogeneous databases based on the multiple second query statements respectively to obtain target data from the multiple heterogeneous databases;

[0139] The data integration processing module 705 is used to perform integration calculations on the target data using data processing expressions to obtain data integration results of heterogeneous databases.

[0140] In some optional implementations, the preset parsing rules include: lexical rules and grammatical combination rules; the syntax tree conversion module 702 specifically includes:

[0141] A lexical analysis unit, configured to perform lexical analysis on the first query statement according to the lexical rules in the preset parsing rules to obtain a lexical result;

[0142] A grammatical combination analysis unit, used to perform grammatical combination analysis on the first query statement according to the grammatical combination rules in the preset parsing rules to obtain a grammatical combination result;

[0143] The syntax tree construction unit is used to construct an abstract syntax tree containing multiple nodes by taking the syntax combination results as tree branches and the lexical results as tree nodes.

[0144] In some optional implementations, the query statement generating module 703 includes:

[0145] A semantic traversal extraction unit is used to perform a depth-first traversal according to the tree branches of the current abstract syntax tree, and obtain a semantic unit in the lexical result of the current tree node each time a tree node is traversed;

[0146] The query statement reconstruction unit is used to assemble multiple semantic units into a second query statement corresponding to the current abstract syntax tree according to the actual grammar rules of the heterogeneous database corresponding to the current abstract syntax tree after obtaining the semantic units in all tree nodes of the current abstract syntax tree, wherein one abstract syntax tree corresponds to one second query statement.

[0147] In some optional implementations, the data query function processing text also includes link data information corresponding to different heterogeneous databases;

[0148] The above device also includes:

[0149] The connection pool establishment module is used to establish a heterogeneous database connection pool according to the connection data information. The database connection pool contains different connection paths with different heterogeneous databases.

[0150] In some optional implementations, the data query execution module 704 includes:

[0151] The query distribution execution unit is used to distribute different second query statements to corresponding heterogeneous databases based on different link paths in the heterogeneous database link pool, and receive target data returned by each heterogeneous database.

[0152] In some optional implementations, the data processing expression is composed of an operator and a heterogeneous data identification variable, wherein one heterogeneous data identification variable corresponds to target data of one heterogeneous database;

[0153] The data integration processing module 705 includes:

[0154] The data integration calculation unit is used to substitute the target data returned by each heterogeneous database into the corresponding heterogeneous data identification variable, and calculate the data integration result of the heterogeneous database through operators.

[0155] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0156] The processing device for heterogeneous data sources of function expressions in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0157] The embodiment of the present invention also provides a computer device having the above Figure 7 The function expression shown is a processing device for heterogeneous data sources.

[0158] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0159] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0160] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0161] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0163] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0164] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0165] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0166] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A method for processing heterogeneous data sources of function expressions, characterized in that: The method comprises: Obtaining a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to a plurality of heterogeneous databases respectively; the first query statements corresponding to the plurality of heterogeneous databases respectively are set based on a preset same custom grammar rule; According to the preset parsing rule corresponding to the preset same custom grammar rule, convert each of the first query statements into a corresponding abstract syntax tree; Traversing all nodes of each abstract syntax tree to generate a second query statement for a heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases; Based on the plurality of second query statements, the corresponding heterogeneous databases are respectively queried to obtain target data from the plurality of heterogeneous databases; The target data is integrated and calculated using the data processing expression to obtain a data integration result of a heterogeneous database.

2. The method according to claim 1, characterized in that The preset parsing rules include: lexical rules and grammatical combination rules; The converting each of the first query statements into a corresponding abstract syntax tree according to the preset parsing rule corresponding to the preset same custom grammar rule specifically includes: Performing lexical analysis on the first query statement according to the lexical rules in the preset parsing rules to obtain a lexical result; Performing grammatical combination analysis on the first query statement according to the grammatical combination rule in the preset parsing rule to obtain a grammatical combination result; The grammar combination results are used as tree branches, and the lexical results are used as tree nodes to construct an abstract syntax tree containing multiple nodes.

3. The method according to claim 2, characterized in that The traversing all nodes of each abstract syntax tree to generate a second query statement for the heterogeneous database corresponding to each abstract syntax tree specifically includes: Perform a depth-first traversal of the tree branches of the current abstract syntax tree, and obtain the semantic unit in the lexical result of the current tree node each time a tree node is traversed; After obtaining the semantic units in all tree nodes of the current abstract syntax tree, multiple semantic units are assembled into a second query statement corresponding to the current abstract syntax tree according to the actual syntax rules of the heterogeneous database corresponding to the current abstract syntax tree, wherein one abstract syntax tree corresponds to one second query statement.

4. The method according to any one of claims 1 to 3, characterized in that: The data query function processing text also includes link data information corresponding to different heterogeneous databases; Before respectively querying the corresponding heterogeneous databases based on the plurality of second query statements to obtain the target data from the plurality of heterogeneous databases, the method further includes: A heterogeneous database link pool is established according to the link data information, and the database link pool contains different link paths with different heterogeneous databases.

5. The method according to claim 4, characterized in that The step of respectively querying the corresponding heterogeneous databases based on the plurality of second query statements to obtain target data from the plurality of heterogeneous databases specifically includes: Based on different link paths in the heterogeneous database link pool, different second query statements are distributed to corresponding heterogeneous databases, and target data returned by each heterogeneous database is received.

6. The method according to claim 5, characterized in that The data processing expression is composed of an operator and a heterogeneous data identification variable, wherein one heterogeneous data identification variable corresponds to target data of one heterogeneous database; The integrating calculation of the target data by using the data processing expression to obtain the data integration result of the heterogeneous database specifically includes: The target data returned by each heterogeneous database is substituted into the corresponding heterogeneous data identification variable, and the data integration result of the heterogeneous database is calculated through operators.

7. A processing device for heterogeneous data sources of function expressions, characterized in that: The device comprises: A query text acquisition module is used to acquire a data query function processing text, wherein the data query function processing text includes first query statements and data processing expressions corresponding to a plurality of heterogeneous databases respectively; the first query statements corresponding to the plurality of heterogeneous databases are set based on a preset same custom grammar rule; A syntax tree conversion module, configured to convert each of the first query statements into a corresponding abstract syntax tree according to a preset parsing rule corresponding to the preset same custom syntax rule; A query statement generation module, used to traverse all nodes of each abstract syntax tree and generate a second query statement for a heterogeneous database corresponding to each abstract syntax tree, wherein the second query statement is set based on actual grammar rules of different heterogeneous databases; A data query execution module, configured to query the corresponding heterogeneous databases based on the plurality of second query statements respectively, and obtain target data from the plurality of heterogeneous databases; The data integration processing module is used to use the data processing expression to perform integration calculation on the target data to obtain the data integration result of the heterogeneous database.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for processing heterogeneous data sources of function expressions described in any one of claims 1-6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for processing heterogeneous data sources of function expressions according to any one of claims 1-6.

10. A computer program product, characterized in that It includes computer instructions, and the computer instructions are used to enable a computer to execute the method for processing heterogeneous data sources of function expressions described in any one of claims 1-6.

Images