Data analysis method and device, computer device and storage medium

Abstract

The application relates to a data analysis method and device, computer equipment, a storage medium and a computer program product. The application relates to the technical field of clouds and can realize data query analysis. The method obtains a data analysis query statement for different data sources; an operator pushdown is performed on a logical plan tree used for representing the data analysis query statement, a pushdown query statement matched with the data analysis query statement is obtained, the pushdown query statement comprises a summary query statement and a plurality of target query statements matched with the data sources one by one; each target query statement is registered to a computing engine, the summary query statement is executed through the computing engine, and a data analysis result corresponding to the data analysis query statement is obtained. The application can analyze data under the condition of multiple data sources through the computing engine, the computing engine can be freely selected according to the data source condition, and the expansibility of joint analysis under the condition of multiple data sources is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data analysis method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of computer technology, cloud technology appears, which is a hosting technology of unifying hardware, software, network and other series of resources in a wide area network or local area network to realize data calculation, storage, processing and sharing. In the cloud technology, database technology plays an important role. In simple terms, a database can be regarded as an electronic file cabinet, i.e. a place for storing electronic files, and users can add, query, update and delete data in the files. A "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs. When data analysis is performed, joint analysis of data stored in different data sources plays a crucial role in fully exploiting the value of data.

[0003] At present, when joint analysis of data stored in different data sources is performed, a computing engine coupling technology is generally used, i.e. a computing engine is selected, and different connectors are customized for different data sources. When joint analysis of data is needed, the computing engine pulls full data from different connectors and performs analysis and calculation in the engine. However, this analysis method needs to be bound to a single computing engine, which is not conducive to expansion and cannot take advantage of the characteristics of different computing engines. SUMMARY

[0004] Therefore, it is necessary to provide a data analysis method and device, computer equipment, computer readable storage medium and computer program product capable of effectively improving the expansibility in the joint analysis process to solve the above technical problems.

[0005] In a first aspect, the present application provides a data analysis method. The method comprises:

[0006] obtaining a data analysis query statement for different data sources;

[0007] pushing down operators for a logical plan tree representing the data analysis query statement to obtain a pushed-down query statement matched with the data analysis query statement, the pushed-down query statement comprising an aggregated query statement and a plurality of target query statements matched one-to-one with the data sources;

[0008] registering each target query statement to a computing engine, executing the aggregated query statement through the computing engine to obtain a data analysis result corresponding to the data analysis query statement.

[0009] In a second aspect, the present application provides a data analysis apparatus. The apparatus comprises:

[0010] a data acquisition module configured to acquire data analysis query statements for different data sources;

[0011] a statement analysis module configured to perform operator pushdown on a logical plan tree used to represent the data analysis query statements, to obtain pushdown query statements matched with the data analysis query statements, wherein the pushdown query statements comprise a summary query statement and a plurality of target query statements matched one-to-one with the data sources;

[0012] a data analysis module configured to register each of the target query statements to a computing engine, to execute the summary query statement through the computing engine, and to obtain a data analysis result corresponding to the data analysis query statement.

[0013] In a third aspect, the present application provides a computer device. The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0014] acquiring data analysis query statements for different data sources;

[0015] performing operator pushdown on a logical plan tree used to represent the data analysis query statements, to obtain pushdown query statements matched with the data analysis query statements, wherein the pushdown query statements comprise a summary query statement and a plurality of target query statements matched one-to-one with the data sources;

[0016] registering each of the target query statements to a computing engine, executing the summary query statement through the computing engine, and obtaining a data analysis result corresponding to the data analysis query statement.

[0017] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0018] acquiring data analysis query statements for different data sources;

[0019] performing operator pushdown on a logical plan tree used to represent the data analysis query statements, to obtain pushdown query statements matched with the data analysis query statements, wherein the pushdown query statements comprise a summary query statement and a plurality of target query statements matched one-to-one with the data sources;

[0020] registering each of the target query statements to a computing engine, executing the summary query statement through the computing engine, and obtaining a data analysis result corresponding to the data analysis query statement.

[0021] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:

[0022] obtaining a data analysis query statement for different data sources;

[0023] performing operator pushdown on a logical plan tree used to represent the data analysis query statement to obtain a pushed query statement matched with the data analysis query statement, the pushed query statement comprising a summary query statement and a plurality of target query statements matched one-to-one with the data sources;

[0024] registering each of the target query statements to a computing engine, executing the summary query statement through the computing engine to obtain a data analysis result corresponding to the data analysis query statement.

[0025] The data analysis method, device, computer device, storage medium and computer program product, wherein the method comprises: obtaining a data analysis query statement for different data sources; performing operator pushdown on a logical plan tree used to represent the data analysis query statement to obtain a pushed query statement matched with the data analysis query statement, the pushed query statement comprising a summary query statement and a plurality of target query statements matched one-to-one with the data sources; registering each of the target query statements to a computing engine, executing the summary query statement through the computing engine to obtain a data analysis result corresponding to the data analysis query statement. After obtaining the data analysis query statement, the present application parses the data analysis query statement into a summary query statement and a plurality of target query statements matched one-to-one with the data sources, and then performs analysis on data in a multi-data-source case through a computing engine, so that the computing engine can be freely selected according to the data source, and the expansibility of joint analysis in a multi-data-source case is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 An application environment diagram of the data analysis method in one embodiment;

[0027] Figure 2 A flowchart of the data analysis method in one embodiment;

[0028] Figure 3 A flowchart of the step of obtaining a pushed query statement matched with the data analysis query statement through operator pushdown in one embodiment;

[0029] Figure 4 A diagram of the change of a logical plan tree before and after operator pushdown rule processing in one embodiment;

[0030] Figure 5A flowchart of the step of generating a pushdown query statement by the visitor pattern in one embodiment;

[0031] Figure 6 A flowchart of the step of constructing a target query statement corresponding to the same data source by a recursive manner in one embodiment;

[0032] Figure 7 A flowchart of the step of generating a target query statement corresponding to a Join in one embodiment;

[0033] Figure 8 A flowchart of the step of obtaining a data analysis result in one embodiment;

[0034] Figure 9 A schematic diagram of the connection manner between the computing engine and the data source in one embodiment;

[0035] Figure 10 A flowchart of the step of registering a target query statement and finally calculating a data analysis result in one embodiment;

[0036] Figure 11 An interaction diagram of creating a MySQL data source in one embodiment;

[0037] Figure 12 A flowchart of the data analysis method in another embodiment;

[0038] Figure 13 A block diagram of the structure of a data analysis apparatus in one embodiment;

[0039] Figure 14 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0041] Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or a local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of a technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item in the future may have its own identification mark and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data need strong system support, which can only be realized through cloud computing. The present application specifically relates to database technology in cloud technology.

[0042] A database can be briefly regarded as an electronic file cabinet, i.e. a place for storing electronic files, and users can add, query, update and delete data in the files. A database is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs. A database management system (DBMS) is a computer software system designed to manage a database, generally with basic functions such as storage, interception, security, backup, etc. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server cluster, mobile phone; or according to the query language they use, such as SQL (Structured Query Language), XQuery; or according to the performance focus, such as maximum size, highest speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, such as supporting multiple query languages at the same time.

[0043] Technical terms involved in the present application also include:

[0044] Data source: a system for storing data, such as traditional relational databases such as Oracle and MySQL, and big data storage systems such as Hive and HBase.

[0045] Joint analysis: data analysis based on data in different data sources.

[0046] JDBC (Java Database Connectivity): Application programming interface to specify how a client program accesses a database, provides methods such as querying and updating data.

[0047] Hive: Both a system for storing big data and a distributed computing engine for big data.

[0048] Spark: A distributed computing engine for big data. Spark has become the industry standard for big data offline processing.

[0049] Presto: An in-memory MPP distributed computing engine.

[0050] RBO (Rule-Based Optimization): According to the optimization rules, the relational expression is converted, and the original expression is cut off. After a series of conversions, the final execution plan is generated.

[0051] The data analysis method provided by the embodiments of the present application can be applied to an application environment as shown in the figure. Figure 1 The terminal 102 communicates with the server 104 through the network, and the server 104 integrates a computing engine. The first data source 106, the second data source 108, and the third data source 110 can store data that needs to be processed by the server 104. The first data source 106, the second data source 108, and the third data source 110 can be integrated on the server 104, or can be placed on the cloud or other servers. When the staff of the terminal 102 needs to integrate data from multiple different data sources for data analysis, a data analysis query statement for different data sources can be sent to the server 104, and the server 104 obtains the data analysis query statement for different data sources. The logical plan tree representing the data analysis query statement is pushed down to the operator to obtain a pushed query statement matched with the data analysis query statement. The pushed query statement includes a summary query statement and multiple target query statements matched with the data sources one by one. Each target query statement is registered to the computing engine, and the summary query statement is executed through the computing engine to obtain a data analysis result corresponding to the data analysis query statement. Then the data analysis result is fed back to the terminal 102. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0052] In one embodiment, asFigure 2 As shown, a data analysis method is provided, and the method is applied to Figure 1 The server 104 in the method is taken as an example for illustration, and the method comprises the following steps:

[0053] In step 201, a data analysis query statement for different data sources is acquired.

[0054] The data source refers to the source of data analyzed by the data analysis method in the present application, and the data source can specifically refer to a system for data storage, such as a traditional relational database Oracle, MySQL, etc., a big data storage system Hive, HBase, etc. Before using the data source, the relevant information of the data source needs to be registered. The data analysis query statement specifically refers to a statement for operating the database, and the data analysis query statement can be used to realize the query and analysis of data in the database. In an embodiment, the data analysis query statement can be a SQL (Structured Query Language) statement, and the data query statement can specify which data sources connected to the server 104 need to be used in the data analysis.

[0055] Specifically, when the staff of the terminal 102 needs to combine the data of multiple data sources to perform data analysis, he can write a corresponding data analysis query statement, and then send the data analysis query statement to the server 104 to start a data analysis process. The server 104 can receive the data analysis query statement, and then start the query according to the data analysis query statement. In the query process, the data analysis query statement for querying different data sources is specifically split into target query statements for different data sources, and then the target query statements are analyzed to ensure the scalability of the data analysis process.

[0056] In step 203, an operator is pushed down to a logical plan tree for representing the data analysis query statement to obtain a pushed-down query statement matched with the data analysis query statement. The pushed-down query statement comprises a summary query statement and multiple target query statements matched with the data sources one by one.

[0057] The logical plan tree is a syntax tree, which is a tree formed when inference is performed according to a certain rule. In the data analysis process, a data analysis query statement can be converted into a syntax tree, and the syntax tree can be analyzed and converted into an execution plan. In the process of big data analysis, the execution plan is divided into a logical execution plan and a physical execution plan for better scalability, fault tolerance, and high availability. The logical plan is divided according to the characteristics of the query, so that the divided logical plan can be distributed to more scalable parallel nodes, and finally the logical execution plan is converted into a physical execution plan for query. The execution process of the data analysis query statement is like a factory assembly line, which is layer by layer and finally gets the desired result. The operator is like a process in the assembly line. For a SQL statement, it includes operators for selecting, projecting, joining, and the like. The operator can include a data source (DataSource), selection (Selection), projection (Projection), join (Join), sorting (Sort), grouping (Aggregation), subquery (Subquery), and the like. Pushdown is a database statement optimization method, which is usually used in the query stage. The operator pushdown can adjust the operator in the logical plan tree. The pushdown query statement includes a summary query statement and a plurality of target query statements matched with the data source. The target query statement is used to query and analyze a single data source. The operator pushdown in the logical plan tree can effectively differentiate the data analysis query statement for different data sources into a plurality of target query statements, thereby realizing query. The summary query statement is used to comprehensively process the query analysis results of the plurality of target query statements.

[0058] Specifically, after obtaining the data analysis query statement, the server 104 converts the data analysis query statement into a logical plan tree, and then performs operator pushdown to split the data analysis query statement for different data sources, to obtain a summary query statement and a plurality of target query statements matched with the data source. Then, the corresponding target query statement can be used to get rid of the connector, thereby realizing data query, and finally the query results of the target query statement are integrated through the summary query statement, so that the final data analysis result is obtained.

[0059] In step 205, each target query statement is registered to a computing engine, and the summary query statement is executed through the computing engine to obtain a data analysis result corresponding to the data analysis query statement.

[0060] In this context, a computing engine is a program specifically designed to process data, such as SQL. Behind SQL is the database's computing engine, but database computation and storage are usually integrated and collectively referred to as the database engine. Computing engines are divided into stream processing engines and batch processing engines. Stream processing engines include Storm, Samza, Flink, and Spark, while batch processing engines include Spark, Hive, Pig, and Flink. Only Apache Flink and Apache Spark support both stream and batch processing.

[0061] Specifically, after obtaining the pushdown query statement, corresponding data query analysis can be performed based on it. During data analysis, each target query statement can be registered with the computing engine. This computing engine can be any engine; there's no need to customize different connectors for different data sources. By registering each target query statement with a computing engine, data query analysis can be performed on the data source corresponding to that target query statement. Finally, by executing the summary query statement through the computing engine, the data analysis results from all data sources can be combined to obtain the data analysis result corresponding to the data analysis query statement.

[0062] The aforementioned data analysis method involves obtaining data analysis query statements for different data sources; pushing down the logical plan tree representing these query statements using operators to obtain pushed-down query statements that match the data analysis query statements. These pushed-down query statements include summary query statements and multiple target query statements that match each data source one-to-one; registering each target query statement with a computing engine; and executing the summary query statements through the computing engine to obtain the data analysis results corresponding to the data analysis query statements. This application, after obtaining the data analysis query statements, parses them into summary query statements and multiple target query statements that match each data source one-to-one. Then, it uses a computing engine to analyze data from multiple data sources. The computing engine can be freely selected based on the data source situation, effectively improving the scalability of joint analysis in multi-data source scenarios.

[0063] In one embodiment, such as Figure 3 As shown, step 203 includes:

[0064] Step 302: Parse and process the data analysis query statement to generate a logical plan tree.

[0065] Step 304: Transform the logical plan tree according to the operator pushdown rules corresponding to the logical plan tree to aggregate operators with the same data source.

[0066] At step 306, according to the transformed logical plan tree, a summary query statement and a plurality of target query statements matched with the data sources are generated.

[0067] The parsing process of the data analysis query statement belongs to the compiler category, including processes such as lexical analysis, syntax and semantic analysis, optimization, and execution code generation. The process of generating the logical plan tree is the process of logical analysis in semantic analysis, which is to analyze what the input data analysis query statement does and what operations it has. Generally, a data analysis query statement has an input and an output. The input data can be obtained after data query analysis processing. Therefore, the logical plan tree generated in the compilation stage can be traversed according to the execution order of the data analysis query statement. In the execution order of the data analysis query statement, what kind of operator is generated when encountering what operation, and the previous expression analysis is called when encountering an expression. The operator pushdown rules specifically include AggregationPushDownRule (aggregation operator pushdown rule), JoinPushDowRule (join operator pushdown rule), UnionPushDownRul (union operator pushdown rule), LimitPushDownRule (limit operator pushdown rule), FilterPushDownRule (filter operator pushdown rule), ProjectPushDownRule (projection operator pushdown rule), OrderbyPushDownRule (sorting operator pushdown rule), etc., which are used to group operators belonging to the same data source together, and different operator pushdown rules are used for different logical plan tree scenarios.

[0068] Specifically, in the process of generating the pushed-down query statement, in order to split the data analysis query statement for different data sources into a plurality of target query statements matched with the data sources, the data analysis query statement needs to be decomposed, and then the operators belonging to the same data source are grouped together. Therefore, the logical plan tree is first generated through parsing, and then the corresponding operator pushdown rule is matched based on the logical plan tree to transform it, so as to group the operators of the same data source in the logical plan tree. Then, through the pushdown process, the summary query statement and the plurality of target query statements matched with the data sources are further generated. In this embodiment, the logical plan tree is transformed through the operator pushdown rule, so as to group the operators of the same data source, which can effectively ensure the effectiveness of the operator pushdown process.

[0069] In one of the embodiments, the method further comprises: matching the logical plan tree by a rule-based optimizer to obtain the operator pushdown rule corresponding to the logical plan tree.

[0070] Among them, the rule-based optimizer is RBO (Rule-Based Optimization), and the rule-based optimizer includes AggregationPushDownRule, JoinPushDowRule, UnionPushDownRule, LimitPushDownRule, FilterPushDownRule, ProjectPushDownRule, OrderbyPushDownRule and various different optimization rules. The logical plan tree can be converted according to the optimization rules to generate the final execution plan.

[0071] Specifically, after obtaining the logical plan tree, rule matching can be performed by the rule-based optimizer. Different logical plan trees correspond to different optimization rules. After the logical plan tree is transformed by the corresponding optimization rule, the operators under the same data source can be effectively summarized to realize operator pushdown. For example, as shown in Figure 4 AggregationPushDown to Union, first match the template of the logical plan tree. When the logical plan tree conforms to Aggregation-Union, the corresponding operator pushdown rule is AggregationPushDownRule. At this time, the Aggregation can be pushed down to the Union operator by AggregationPushDownRule, and a new Aggregation operator is generated to ensure the correctness of the result. In this embodiment, the rule matching of the logical plan tree is performed by the rule-based optimizer, and the corresponding operator pushdown rule can be effectively found to realize operator pushdown and obtain the corresponding pushdown query statement.

[0072] In one of the embodiments, as shown in Figure 5 The step 306 includes:

[0073] Step 502, based on the transformed logical plan tree, the abstract syntax tree is constructed layer by layer by the visitor pattern.

[0074] Step 504, based on the abstract syntax tree, a plurality of target query statements matched with the data source are recursively generated.

[0075] Step 506, the summary query statement is generated according to the target query statement.

[0076] Among them, the visitor mode, that is, the visitor mode, indicates an operation on each element in an object structure, which enables you to define new operations on these elements without changing the classes of the elements. For example, you visit your friend's home, you are the visitor, your friend receives your visit, and then you make a judgment on the description of your friend, which is the visitor mode. The visitor mode can encapsulate some operations on each element of a certain data structure, and it can define new operations on these elements without changing the data structure. Abstract syntax tree can be understood as an organization form of database statement. For example, for SQL statement, the corresponding abstract syntax tree is SqlNode, and SqlNode is the organization form of dynamic SQL configuration in the program. Each XML Node is parsed into the corresponding SqlNode object. For recursion, the programming skill of calling itself is called recursion. Recursion is widely used in programming languages as an algorithm. A process or function has a direct or indirect call to itself in its definition or description. It usually translates a large and complex problem into a smaller problem similar to the original problem to solve. The recursive strategy only needs a small amount of program to describe the repeated calculation required in the problem solving process, which greatly reduces the code amount of the program. The ability of recursion is to define an infinite set of objects with limited statements. Generally speaking, recursion needs boundary conditions, recursive advance segment and recursive return segment. When the boundary condition is not met, recursion advances; when the boundary condition is met, recursion returns. In this application, the final target query statement is mainly constructed by using the recursive mode.

[0077] Specifically, the process of generating the target query statement can be regarded as the process of reversing the operators of the same data source into a database query statement, after the logical plan tree is transformed by the operator down-push rule to group the operators of the same data source. Since the operators of the same data source have been grouped together, the operators of the same data source can be reversed into a database query statement at this time. Through the visitor mode, a complete abstract syntax tree can be constructed from bottom to top layer by layer based on the operators of the same data source, and then a complete target query statement for a single data source is generated by using the recursive mode. Then, when the target query statements corresponding to all data sources are obtained, the target query statements can be used to generate a summary query statement, and the summary query statement is used to comprehensively process the query results of the target query statements to obtain the final data analysis result. In this embodiment, the abstract syntax tree is constructed layer by layer through the visitor mode, so that multiple target query statements matched with the data sources can be effectively constructed, which can effectively guarantee the effectiveness of the target query statement and the accuracy of the data query result.

[0078] In one of the embodiments, as Figure 6As shown, step 502 includes:

[0079] Step 601, based on the abstract syntax tree, the corresponding child node query statement of the data source is generated by accessing mode.

[0080] Step 603, according to the corresponding child node query statement of the same data source, the corresponding target query statement of the same data source is constructed by recursive mode.

[0081] Among them, the abstract syntax tree of the same data source constructed by the visitor mode layer by layer can contain multiple child nodes, and the child node query statement is the query statement corresponding to a single child node.

[0082] Specifically, in constructing a target query statement, first, the abstract syntax tree corresponding to a single data source needs to be determined, based on the abstract syntax tree, the corresponding child node query statement of the data source is generated by accessing mode; finally, according to the corresponding child node query statement of the same data source, the corresponding target query statement of the same data source is constructed by recursive mode. As Figure 7 As shown, in one of the embodiments, the data analysis query statement is specifically SQL statement. Taking Join as an example, when the corresponding target query statement of Join needs to be generated, the corresponding SQL statement is generated by accessing mode based on the left and right child nodes of Join, and finally the final Join SQL is constructed. The corresponding pseudo code of pushdown SQL generation is as follows:

[0083] visit(JdbcJoin joinNode)

[0084] 1:SqlNode leftSql=visitChild(joinNode.leftNode)

[0085] 2:SqlNode rightSql=visitChild(joinNode.rightNode)

[0086] 3:return JoinSqlNode(leftSql,rightSql,joinNode.joinType)

[0087] In this embodiment, the corresponding target query statement of the same data source is effectively constructed by constructing the corresponding target query statement of the same data source through recursive mode based on the corresponding child node query statement of the same data source, which guarantees the accuracy of data analysis and processing.

[0088] In one of the embodiments, as Figure 8 As shown, step 205 includes:

[0089] Step 801, register the pushed-down query statement in the form of view to the computing engine, and obtain the data query result corresponding to each data source.

[0090] Step 803, execute the summary query statement by the computing engine to perform data analysis on the data query result corresponding to each data source, and obtain the data analysis result corresponding to the data analysis query statement.

[0091] The view refers to a view in a computer database, which is a virtual table, and its content is defined by a query. Like a real table, the view contains a series of columns with names and row data. However, the view does not exist in the form of a stored data value set in the database. The row and column data are from the table referenced by the query that defines the view, and are generated dynamically when the view is referenced.

[0092] Specifically, the pushed-down target query statements of different data sources can be registered in the form of view to the computing engine of the server, so as to obtain the query results corresponding to the target query statements by the computing engine, and finally, the query results of the target query statements can be summarized by using the computing engine to execute a summary query statement, so as to obtain a final data analysis result. In one embodiment, as shown in Figure 9 Each data source connected with the server 104 provides a standard JDBC interface. The calculation of the data source is pushed down to the data source through the JDBC, and the data analysis result is pulled through the JDBC after the calculation is completed. Based on the results pulled from different data sources JDBC, the computing engine completes the final joint calculation. As shown in Figure 10 As shown in the figure, assuming that an SQL query involves N JDBC data sources, at least N+1 SQL statements (3 SQL statements in the figure, corresponding to 3 views) will be generated, in which N SQLs need to be pushed down to the data source for execution (one SQL corresponds to one data source, and each SQL is mapped to a view of the computing engine), and the last one is responsible by the computing engine to summarize the results of each view (i.e. the result of the pushed-down calculation) and return the result. In this embodiment, by registering the pushed-down query statement in the form of view to the computing engine, the calculation of the pushed-down query statement can be effectively completed in the computing engine, so as to ensure the effectiveness of the final data analysis.

[0093] The application also provides an application scenario applying the above-mentioned data analysis method. Specifically, the data analysis method is applied in the application scenario as follows:

[0094] The user can store, query and analyze the commodity sales information data through the database service, and the user can store the data through multiple different data sources (databases). Before the user uses a data source, the user needs to register the related information of the data source. For example Figure 11 An interaction diagram for creating a MySQL data source is shown. After the creation, the data in the data source can be queried through the name of the data source (datasource1). When the user needs to perform a data analysis process on the data of multiple data sources, such as analyzing the sales situation based on the sales data stored in multiple databases, the user can write an SQL statement for querying and analyzing the data of multiple data sources on the terminal, and then send the SQL statement to the server to perform the corresponding data query and analysis through the server. The server processes the SQL statement to perform the data analysis process, as shown in Figure 12 First, the server needs to parse the obtained SQL statement to generate a logical plan tree, and then apply the operator pushdown rule to push down the SQL statement to reversely generate N+1 pushdown SQL statements, where N is the number of data sources, that is, it includes multiple SQL statements matched with the data sources, and in addition, it includes a summary SQL statement. Then, the server can create N JDBC views in the computing engine according to the pushdown SQL statement, and submit the summary SQL statement to the computing engine, so as to perform the final data analysis through the computing engine to obtain the data analysis result corresponding to the SQL statement. The process of parsing the obtained SQL statement to generate a logical plan tree, and then applying the operator pushdown rule to push down the SQL statement to reversely generate N+1 pushdown SQL statements specifically includes: parsing and processing the SQL statement to generate a logical plan tree; transforming the logical plan tree according to the operator pushdown rule corresponding to the logical plan tree to aggregate the operators of the same data source; generating the pushdown SQL statement according to the transformed logical plan tree. The operator pushdown rule is obtained by matching the logical plan tree through the rule-based optimizer. The generation process of the pushdown SQL statement specifically includes: constructing an abstract syntax tree layer by layer through the visitor pattern based on the transformed logical plan tree; generating a sub-node query statement corresponding to the data source through the access mode based on the abstract syntax tree; constructing a target query statement corresponding to the same data source through the recursive mode according to the sub-node query statement corresponding to the same data source; generating a summary query statement according to the target query statement. Finally, the process of obtaining the analysis result needs to register the pushdown query statement in the form of a view to the computing engine to obtain the data query result corresponding to each data source; the computing engine executes the summary query statement to perform data analysis on the data query result corresponding to each data source to obtain the data analysis result corresponding to the data analysis query statement.

[0095] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0096] Based on the same inventive concept, the embodiments of the present application also provide a data analysis device for implementing the above-mentioned data analysis method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data analysis device embodiments provided below can refer to the limitations of the data analysis method in the above text, which will not be repeated here.

[0097] In one embodiment, as shown in Figure 13 a data analysis device is provided, comprising:

[0098] The data acquisition module 1302 is configured to acquire data analysis query statements for different data sources.

[0099] The statement analysis module 1304 is configured to perform operator pushdown on the logical plan tree representing the data analysis query statement to obtain a pushdown query statement matching the data analysis query statement, the pushdown query statement including a summary query statement and a plurality of target query statements one-to-one matched with the data sources.

[0100] The data analysis module 1306 is configured to register each target query statement to a computing engine, execute the summary query statement through the computing engine, and obtain a data analysis result corresponding to the data analysis query statement.

[0101] In one embodiment, the statement analysis module 1304 is specifically configured to: perform analysis processing on the data analysis query statement to generate a logical plan tree; perform transformation processing on the logical plan tree according to an operator pushdown rule corresponding to the logical plan tree to summarize operators of the same data source; and generate the summary query statement and the plurality of target query statements one-to-one matched with the data sources according to the transformed logical plan tree.

[0102] In one embodiment, the statement analysis module 1304 is further configured to: match the logical plan tree through a rule-based optimizer to obtain the operator pushdown rule corresponding to the logical plan tree.

[0103] In one embodiment, the statement parsing module 1304 is further configured to: construct an abstract syntax tree layer by layer based on the transformed logical plan tree using the visitor pattern; recursively generate multiple target query statements that match the data source one by one based on the abstract syntax tree; and generate a summary query statement based on the target query statements.

[0104] In one embodiment, the statement parsing module 1304 is further configured to: generate a child node query statement corresponding to the data source based on an abstract syntax tree and through an access method; and construct a target query statement corresponding to the same data source through a recursive method based on the child node query statement corresponding to the same data source.

[0105] In one embodiment, the data analysis module 1306 is specifically used to: register the pushdown query statement to the computing engine in the form of a view, obtain the data query results corresponding to each data source; execute the summary query statement through the computing engine to perform data analysis on the data query results corresponding to each data source, and obtain the data analysis results corresponding to the data analysis query statement.

[0106] Each module in the aforementioned data analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0107] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data related to data analysis. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a data analysis method.

[0108] Those skilled in the art will understand that Figure 14The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In an embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0110] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0111] In an embodiment, a computer program product or computer program is provided, including computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the steps in the above method embodiments.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0114] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0115] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data analysis method, characterized by, The method comprises: acquiring a data analysis query statement for different data sources; pushing down operators of a logical plan tree representing the data analysis query statement to obtain a pushed-down query statement matched with the data analysis query statement, the pushed-down query statement comprising an aggregated query statement and a plurality of target query statements matched one-to-one with the data sources, the target query statements being used for query analysis on a single matched data source, and the aggregated query statement being used for comprehensive processing of query analysis results of the plurality of target query statements; registering each target query statement to a computing engine, executing the aggregated query statement through the computing engine, and obtaining a data analysis result corresponding to the data analysis query statement; the pushing down of operators of the logical plan tree representing the data analysis query statement to obtain the pushed-down query statement matched with the data analysis query statement comprises: performing parsing processing on the data analysis query statement to generate a logical plan tree; transforming the logical plan tree according to an operator push-down rule corresponding to the logical plan tree to aggregate operators of the same data source; based on the transformed logical plan tree, constructing an abstract syntax tree layer by layer through a visitor mode, generating a plurality of target query statements matched one-to-one with the data sources based on the abstract syntax tree, and generating an aggregated query statement according to the target query statements.

2. The method of claim 1, wherein, The method further comprises: matching the logical plan tree through a rule-based optimizer to obtain an operator push-down rule corresponding to the logical plan tree.

3. The method of claim 1, wherein, The generating of a plurality of target query statements matched one-to-one with the data sources based on the abstract syntax tree comprises: generating a child node query statement corresponding to a data source through a visitor mode based on the abstract syntax tree; constructing a target query statement corresponding to the same data source through a recursive mode according to the child node query statements corresponding to the same data source.

4. The method of claim 1, wherein, The registering of each target query statement to the computing engine, the executing of the aggregated query statement through the computing engine, and the obtaining of the data analysis result corresponding to the data analysis query statement comprise: registering the pushed-down query statement to the computing engine in the form of a view to obtain a data query result corresponding to each data source; performing data analysis on the data query result corresponding to each data source through the computing engine to obtain a data analysis result corresponding to the data analysis query statement.

5. A data analysis device, characterized by, The apparatus comprises: a data acquisition module configured to acquire a data analysis query statement for different data sources; a statement parsing module configured to push down operators of a logical plan tree representing the data analysis query statement to obtain a pushed-down query statement matched with the data analysis query statement, the pushed-down query statement comprising an aggregated query statement and a plurality of target query statements matched one-to-one with the data sources, the target query statements being used for query analysis on a single matched data source, and the aggregated query statement being used for comprehensive processing of query analysis results of the plurality of target query statements; The data analysis module is configured to register the target query statements to a computing engine, execute the summary query statements through the computing engine, and obtain data analysis results corresponding to the data analysis query statements. The statement analysis module is specifically configured to: analyze the data analysis query statements to generate a logical plan tree; transform the logical plan tree according to an operator pushdown rule corresponding to the logical plan tree to aggregate operators of the same data source; construct an abstract syntax tree layer by layer through a visitor mode based on the transformed logical plan tree; generate a plurality of target query statements one-to-one matched with the data sources based on the abstract syntax tree; and generate summary query statements according to the target query statements.

6. The apparatus of claim 5, wherein, The statement analysis module is further configured to: match the logical plan tree through a rule-based optimizer to obtain an operator pushdown rule corresponding to the logical plan tree.

7. The apparatus of claim 5, wherein, The statement analysis module is further configured to: generate sub-node query statements corresponding to the data sources through a visitor mode based on the abstract syntax tree; and construct target query statements corresponding to the same data source through a recursive mode according to the sub-node query statements corresponding to the same data source.

8. The apparatus of claim 5, wherein, The data analysis module is specifically configured to: register the pushdown query statements to the computing engine in the form of views to obtain data query results corresponding to the data sources; and execute the summary query statements through the computing engine to perform data analysis on the data query results corresponding to the data sources to obtain data analysis results corresponding to the data analysis query statements. The data analysis module is specifically configured to: register the pushdown query statements to the computing engine in the form of views to obtain data query results corresponding to the data sources; and execute the summary query statements through the computing engine to perform data analysis on the data query results corresponding to the data sources to obtain data analysis results corresponding to the data analysis query statements. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 4.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 4.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 4.

Images