Data analysis method, device, equipment and storage medium

Abstract

The present application discloses a data analysis method, which includes: determining at least one data source to be analyzed used in a target intelligent analysis process; generating a target directed acyclic graph corresponding to at least one data source to be analyzed based on the at least one data source to be analyzed; starting from the path starting point node of the target path in the target directed acyclic graph, grouping the path nodes in the target path according to the path direction of the target path based on the relationship that they belong to the same data source and are adjacent to each other, thereby obtaining at least one group of path nodes of the target path; from the target data source corresponding to each group of path nodes of the target path, obtaining and analyzing the corresponding node data of each group of path nodes according to the path direction of the target path, and determining the execution result of the target path, thereby reducing network IO overhead and effectively improving the analysis efficiency of BI analysis. The present application also discloses a data analysis device, equipment, and storage medium.

Description

Technical Field

[0001] The present application relates to the field of data statistics, and in particular to a data analysis method, apparatus, device and storage medium. Background Art

[0002] The process of analyzing transaction data during a transaction can be called Business Intelligence (BI) analysis. In traditional transaction processes, the types of data sources are relatively simple. However, with the popularization of big data technology, the types of data sources are becoming increasingly diverse, and traditional BI analysis tools must gradually adapt to big data query and analysis in the context of multiple different types of data sources. Currently, a unified Structured Query Language (SQL) query engine is commonly used to implement cross-data source queries and analysis between the same or different types of data sources. The specific implementation process can be to proxy and parse SQL across data sources, generate an Abstract Syntax Tree (AST), convert the AST into a logical execution plan, and then generate a physical execution plan. Each sub-SQL is executed according to the physical execution plan, and the query results of each sub-SQL are aggregated and returned to obtain the final result.

[0003] However, in the current implementation process described above, when executing a certain SQL statement, the same data source needs to be accessed multiple times, which severely increases the network input / output (IO) overhead. This defect urgently needs to be optimized.

[0004] Application Contents

[0005] In order to solve the above technical problems, this application hopes to provide a data analysis method, device, equipment and storage medium, which solves the current problem of large network IO overhead, reduces network IO overhead, and effectively improves the analysis efficiency of BI analysis.

[0006] The technical solution of this application is achieved as follows:

[0007] In a first aspect, a data analysis method is provided, comprising:

[0008] Determine at least one data source to be analyzed used in the target intelligent analysis process;

[0009] Based on the at least one data source to be analyzed, generating a target directed acyclic graph corresponding to the at least one data source to be analyzed; wherein the target directed acyclic graph includes k paths, where k is an integer greater than or equal to 1;

[0010] Starting from a path starting point node of a target path in the target directed acyclic graph, grouping the path nodes in the target path according to the path direction of the target path based on the relationship that the path nodes belong to the same data source and are adjacent to each other, to obtain at least one group of path nodes of the target path; wherein the target path is used to represent each of the k paths;

[0011] Obtaining, from a target data source corresponding to each group of path nodes of the target path according to the path direction of the target path, node data corresponding to each group of path nodes and analyzing the data to determine an execution result of the target path; wherein the target data source corresponding to each group of path nodes of the target path belongs to the at least one data source to be analyzed;

[0012] The execution results of the k paths are summarized to obtain a target analysis result for target intelligent analysis.

[0013] In a second aspect, a data analysis device is provided, comprising: a determination unit, a generation unit, a processing unit, and a summary unit; wherein:

[0014] The determining unit is configured to determine at least one data source to be analyzed used in the target intelligent analysis process;

[0015] The generating unit is configured to generate a target directed acyclic graph corresponding to the at least one data source to be analyzed based on the at least one data source to be analyzed; wherein the target directed acyclic graph includes k paths, where k is an integer greater than or equal to 1;

[0016] The processing unit is configured to group the path nodes in the target path in the target directed acyclic graph, starting from a path starting point node of the target path, according to the path direction of the target path, based on the relationship that the path nodes belong to the same data source and are adjacent, to obtain at least one group of path nodes of the target path; wherein the target path is used to represent each of the k paths;

[0017] The processing unit is further configured to obtain, from a target data source corresponding to each group of path nodes of the target path, node data corresponding to each group of path nodes according to the path direction of the target path, and perform analysis to determine an execution result of the target path; wherein the target data source corresponding to each group of path nodes of the target path belongs to the at least one data source to be analyzed;

[0018] The summarizing unit is used to summarize the execution results of the k paths to obtain a target analysis result for target intelligent analysis.

[0019] According to a third aspect, a data analysis device includes a memory, a processor, and a communication bus; wherein:

[0020] The memory is used to store executable instructions;

[0021] The communication bus is used to realize the communication connection between the processor and the memory;

[0022] The processor is used to execute the data analysis program stored in the memory to implement the steps of the data analysis method as described in any one of the above items.

[0023] In a fourth aspect, a storage medium is provided, wherein a data analysis program is stored on the storage medium, and when the data analysis program is executed by a processor, the steps of the data analysis method as described in any one of the above items are implemented.

[0024] Embodiments of the present application provide a data analysis method, apparatus, device, and storage medium. The method determines at least one data source to be analyzed used in a target intelligent analysis process, generates a target directed acyclic graph corresponding to at least one data source to be analyzed based on the at least one data source to be analyzed, and then, starting from the path starting point node of the target path in the target directed acyclic graph, grouping the path nodes in the target path according to the path direction of the target path based on the relationship that the path nodes belong to the same data source and are adjacent to each other, obtains at least one group of path nodes of the target path, obtains node data of each group of path nodes corresponding to the target data source according to the path direction of the target path, and analyzes the data, determines the execution result of the target path, and summarizes the execution results of k paths to obtain the target analysis result for the target intelligent analysis. In this way, when performing cross-data source query and analysis, a corresponding target directed acyclic graph is generated for at least one data source to be analyzed, and then the path nodes in each target path in the target directed acyclic graph are grouped to obtain at least one group of path nodes of the target path, and the node data of the group of path nodes is obtained from the target data source corresponding to each group of path nodes and analyzed to obtain the execution result of each target path. Finally, the execution results of k paths are summarized to obtain the target analysis result for target intelligent analysis, which realizes the technical solution of simultaneously extracting the node data of adjacent nodes belonging to the same target data source from the target data source, solves the current problem of large network IO overhead, reduces the network IO overhead, and effectively improves the analysis efficiency of BI analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a data analysis method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of another data analysis method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a target directed acyclic graph provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of a data analysis device provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a data analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0031] The embodiment of the present application provides a data analysis method, referring to Figure 1 As shown, the method is applied to a data analysis device, and the method comprises the following steps:

[0032] Step 101: Determine at least one data source to be analyzed used in a target intelligent analysis process.

[0033] In an embodiment of the present application, target intelligent analysis can be, for example, a process of performing intelligent analysis on a certain business activity to obtain the corresponding value generated by the data in the business activity, or a process of analyzing other data. In current business activities, due to the popularity of big data technology, multiple data sources appear in the target business intelligent analysis process. Therefore, when performing the target business intelligent analysis process, at least one data source to be analyzed included in the target business intelligent analysis is first determined. The data analysis device can be a computer device or a server device.

[0034] Step 102: Based on at least one data source to be analyzed, generate a target directed acyclic graph corresponding to at least one data source to be analyzed.

[0035] The target directed acyclic graph includes k paths, where k is an integer greater than or equal to 1.

[0036] In an embodiment of the present application, in mathematics, especially in graph theory and computer science, a directed acyclic graph refers to a directed graph without loops. By analyzing at least one data source to be analyzed, the relationship between the structure in each data source to be analyzed and at least one data source to be analyzed can be determined, thereby generating a target directed acyclic graph corresponding to at least one data source to be analyzed. The relationship between at least one data source to be analyzed can be a logical relationship before and after execution, for example, a chronological order. The target path is a complete path in the target directed acyclic graph starting from the path starting node to the corresponding path ending node.

[0037] Step 103: Starting from the path starting point node of the target path in the target directed acyclic graph, group the path nodes in the target path according to the path direction of the target path and based on the relationship that the path nodes belong to the same data source and are adjacent to each other, to obtain at least one group of path nodes of the target path.

[0038] The target path is used to represent each of the k paths.

[0039] In an embodiment of the present application, path nodes in a target path in a target directed acyclic graph are grouped. The grouping method is to divide adjacent path nodes in the target path that belong to the same data source into a group of path nodes. In this way, the target path can be composed of at least one group of path nodes. The path starting point node of the target path is the path node at which the path of the target path begins.

[0040] Step 104: From the target data source corresponding to each group of path nodes of the target path, obtain the node data corresponding to each group of path nodes according to the path direction of the target path and analyze the data to determine the execution result of the target path.

[0041] The target data source corresponding to each group of path nodes of the target path belongs to at least one data source to be analyzed.

[0042] In an embodiment of the present application, each target path in the target directed acyclic graph needs to be processed separately in parallel. In order to reduce the number of accesses to the target data source, the node data of all path nodes of each group of path nodes can be obtained from the target data source corresponding to the group of path nodes at one time. Then, the node data of each corresponding path node is analyzed according to the path direction of the group of path nodes in the target path, until the execution result of each target path is finally obtained. The analysis of each target path in the target directed acyclic graph can be performed in parallel.

[0043] Step 105: Summarize the execution results of the k paths to obtain target analysis results for target intelligent analysis.

[0044] In this embodiment of the present application, the endpoint nodes of the k paths are the same node. Therefore, the execution results obtained by analyzing the k paths can be summarized based on the endpoint nodes of the k paths to obtain the target analysis results for the target intelligent analysis. The endpoint node is the last path node of each path.

[0045] An embodiment of the present application provides a data analysis method, which determines at least one data source to be analyzed used in a target intelligent analysis process, generates a target directed acyclic graph corresponding to at least one data source to be analyzed based on the at least one data source to be analyzed, and then, starting from the path starting point node of the target path in the target directed acyclic graph, grouping the path nodes in the target path according to the path direction of the target path based on the relationship that the path nodes belong to the same data source and are adjacent, obtains at least one group of path nodes of the target path, obtains node data of each group of path nodes corresponding to the target data source according to the path direction of the target path and analyzes the data, determines the execution result of the target path, and summarizes the execution results of k paths to obtain the target analysis result for the target intelligent analysis. In this way, when performing cross-data source query and analysis, a corresponding target directed acyclic graph is generated for at least one data source to be analyzed, and then the path nodes in each target path in the target directed acyclic graph are grouped to obtain at least one group of path nodes of the target path, and the node data of the group of path nodes is obtained from the target data source corresponding to each group of path nodes and analyzed to obtain the execution result of each target path. Finally, the execution results of k paths are summarized to obtain the target analysis result for target intelligent analysis, which realizes the technical solution of simultaneously extracting the node data of adjacent nodes belonging to the same target data source from the target data source, solves the current problem of large network IO overhead, reduces the network IO overhead, and effectively improves the analysis efficiency of BI analysis.

[0046] Based on the above embodiments, the embodiments of the present application provide a data analysis method, referring to Figure 2 As shown, the method is applied to a data analysis device, and the method comprises the following steps:

[0047] Step 201: Determine at least one data source to be analyzed used in a target intelligent analysis process.

[0048] In an embodiment of the present application, when determining that target intelligent analysis needs to be performed, the data analysis device determines at least one data source to be analyzed involved in the target intelligent analysis.

[0049] Step 202: Generate an abstract syntax tree (AST) based on at least one data source to be analyzed.

[0050] In embodiments of the present application, an Abstract Syntax Tree (AST), or simply a Syntax Tree, is an abstract representation of the grammatical structure of source code. It represents the grammatical structure of a programming language in a tree-like form, with each node in the tree representing a structure in the source code. A data analysis device analyzes at least one data source to be analyzed, determines the logical structure within each data source to be analyzed, and generates an AST based on the dependencies between the logical structure and the data source to be analyzed.

[0051] In other embodiments of the present application, step 202 may be implemented by steps a11 to a14:

[0052] Step a11: Determine reference dimension parameters and reference metric parameters of each of at least one data source to be analyzed.

[0053] In an embodiment of the present application, a model is established for the data in each data source to be analyzed. The specific process includes determining information such as each dimensional parameter, i.e., a reference dimensional parameter, and each metric parameter, i.e., a reference metric parameter, in each data source to be analyzed.

[0054] Step a12: Based on the reference dimension parameters and reference metric parameters of each data source to be analyzed, classify and encapsulate the data in each data source to be analyzed to obtain encapsulated data of each data source to be analyzed.

[0055] In an embodiment of the present application, the data corresponding to each reference dimension parameter of each data source to be analyzed is classified, packaged and stored, and each reference measurement parameter of each data source to be analyzed is classified, packaged and stored to obtain the packaged data of each data source to be analyzed.

[0056] Step a13: Obtain target processing parameters.

[0057] Among them, the target processing parameters include target dimension parameters and / or target measurement parameters.

[0058] In an embodiment of the present application, the target dimension parameter and the target measurement parameter may be input by the user through an input device, for example, the input process may be implemented through a device such as a keyboard, a mouse, or a microphone. The target dimension parameter includes at least one dimension parameter, and / or the target measurement parameter includes at least one measurement parameter. That is, in some application scenarios, the user may select only the dimension parameter as the target dimension parameter for business intelligence analysis, or may select only the measurement parameter as the target measurement parameter for intelligent analysis. The target processing parameter may include at least one processing parameter, that is, a parameter obtained by at least one arbitrary combination of the target dimension parameter and / or the target measurement parameter.

[0059] Step a14: Based on the target processing parameters, the encapsulated data of at least one data source to be analyzed is processed to obtain an AST.

[0060] In an embodiment of the present application, the encapsulated data of at least one determined data source to be analyzed is analyzed and processed using target processing parameters to generate an AST.

[0061] In other embodiments of the present application, step a14 can be implemented by steps a141 to a142:

[0062] Step a141: Generate an SQL query statement based on the target processing parameters according to the preset SQL statement rules.

[0063] Step a142: Using an open source SQL parsing tool, based on SQL query statements, parse the encapsulated data of at least one data source to be analyzed to obtain an AST.

[0064] Step 203: Generate a target directed acyclic graph based on the AST.

[0065] In an embodiment of the present application, each node in the AST is preprocessed to generate a target directed acyclic graph.

[0066] In other embodiments of the present application, step 203 may be implemented by steps b11 to b12:

[0067] Step b11: Verify the grammatical structure object in each node of the AST.

[0068] In an embodiment of the present application, the process of preprocessing each node in the AST includes: verifying the grammatical structure object in each node of the AST, and the specific verification content at least includes determining whether a collection of database objects (XML Schema, Schema for short), fields, functions, etc. exists, and / or whether the SQL statement is legal, etc.

[0069] Step b12: If the grammatical structure object in each node of the AST passes verification, merge the same subtrees in the AST to obtain the target directed acyclic graph.

[0070] In the embodiment of the present application, after the grammatical structure objects in each node of the AST are verified, the same subtrees in the AST are merged to obtain the target directed acyclic graph. In this way, the target directed acyclic graph can reflect the dependency relationship between multiple data sources and the data represented by each node.

[0071] Step 204: Starting from the path starting point node of the target path in the target directed acyclic graph, group the path nodes in the target path according to the path direction of the target path and based on the relationship that the path nodes belong to the same data source and are adjacent to each other, to obtain at least one group of path nodes of the target path.

[0072] Among them, the target path is used to represent each path in the k paths;

[0073] In an embodiment of the present application, after the data analysis device executes step 204, it can choose to execute steps 205 to 206, or steps 207 to 215. Among them, if the target path of the target directed acyclic graph includes a group of path nodes, choose to execute steps 205 to 206; if the target path of the target directed acyclic graph includes at least two groups of path nodes, choose to execute steps 207 to 215. It should be noted that in the target directed acyclic graph, there are some paths in which all path nodes except the last path endpoint node belong to one target data source, and some paths except the last path endpoint node belong to at least two target data sources. In this way, the data analysis device can simultaneously choose to execute corresponding steps 205 to 206 and steps 207 to 215 for different paths.

[0074] Step 205: If the target path includes a group of path nodes, obtain first node data of the group of path nodes from a target data source corresponding to the group of path nodes.

[0075] A group of path nodes includes path nodes in the target path except the path end node.

[0076] In an embodiment of the present application, a path including three path nodes A, B and C is taken as an example for explanation. The path is A→B→C. If the two path nodes A and B belong to the same target data source, it can be determined that A and B belong to the same group of path nodes. Correspondingly, the data corresponding to the two path nodes A and B can be obtained from the target data source at the same time to obtain the first node data.

[0077] Step 206: According to the path direction of the target path, the target task of each path node in a group of path nodes is executed in sequence based on the first sub-data corresponding to each path node in the first node data to obtain the execution result of the target path.

[0078] In an embodiment of the present application, the target task processing of path node A is performed on the data corresponding to path node A in the first node data to obtain the sub-execution result of path node A, and then the sub-execution result of path node A and the data corresponding to path node B in the first node data are processed with the target task processing of path node B to obtain the sub-execution result of path node B, and the sub-execution result of path node B is used as the execution result of the path A→B→C.

[0079] Step 207: If the target path includes at least two groups of path nodes, determine the target data source corresponding to the first group of path nodes including the path starting point node of the target path according to the path direction of the target path.

[0080] In an embodiment of the present application, a path including 6 path nodes E, F, G, H, I and J is taken as an example for explanation. The target path is E→F→G→H→I→J, the path starting node E belongs to the first target data source, and the path node adjacent to the path starting node E is F. It is judged whether the path node F belongs to the first target data source. If the path node F does not belong to the first target data source, it can be determined that E is a group of path nodes, that is, it corresponds to the first group of path nodes.

[0081] Step 208: Obtain the second node data of the first group of path nodes from the target data source corresponding to the first group of path nodes.

[0082] In the embodiment of the present application, the second node data of the first group of path nodes E is obtained from the target data source corresponding to the first group of path nodes.

[0083] Step 209: Execute the target task corresponding to each path node in the first group of path nodes based on the second sub-data corresponding to each path node in the second node data according to the path direction of the target path, and obtain a first sub-execution result.

[0084] In an embodiment of the present application, since there is only one path node E in the first group of path nodes, the second node data at this time is the second sub-data corresponding to the path node E. Correspondingly, the second sub-data is analyzed according to the target task of the path node E to obtain the first sub-execution result.

[0085] Step 210: Determine, in the path direction of the target path, target data sources corresponding to the second group of path nodes adjacent to the target data sources corresponding to the first group of path nodes.

[0086] In this embodiment of the present application, assuming that path node F belongs to the second target data source, path node G adjacent to path node F also belongs to the second target data source, and path node H adjacent to path node G also belongs to the second target data source, path nodes F, G, and H can be determined to belong to the second group of path nodes. Path node I adjacent to path node H belongs to the first target data source, and path node I can be determined to belong to the third group of path nodes. It should be noted that although path node J also belongs to the first target data source, it is not adjacent to path node E and therefore does not belong to the first group of path nodes.

[0087] Step 211: Acquire third node data of the second group of path nodes from a target data source corresponding to the second group of path nodes.

[0088] In an embodiment of the present application, the third node data of the second reference path nodes F, G and H are obtained from the second target data source.

[0089] Step 212: According to the path direction of the target path, based on the first sub-execution result and the third sub-data corresponding to each path node in the third node data, execute the target task corresponding to each path node in the second group of path nodes to obtain a second sub-execution result.

[0090] In an embodiment of the present application, based on the target path, it can be determined that there is a logical relationship between path node F and path node E. Therefore, the first sub-execution result and the third sub-data corresponding to path node F can be analyzed based on the target task corresponding to path node F to obtain the analysis result of path node F, and then the analysis result of path node F and the third sub-data corresponding to path node G can be analyzed based on the target task corresponding to path node G to obtain the analysis result of path node G. The analysis result of path node G and the third sub-data corresponding to path node H can be analyzed based on the target task corresponding to path node H to obtain the analysis result corresponding to path node H, and the analysis result corresponding to path node F is used as the second sub-execution result.

[0091] In this embodiment of the present application, after executing step 212, the data analysis device selects to execute step 213 or steps 214-215. If the second group of path nodes includes path nodes adjacent to the destination node in the target path, step 213 is executed. If the second group of path nodes includes at least one path node other than the destination node in the target path, steps 214-215 are executed.

[0092] Step 213: If the second group of path nodes includes a path node in the target path that is adjacent to the path end node of the target path, determine that the execution result of the target path is the second sub-execution result.

[0093] Step 214: If the second group of path nodes includes at least one path node other than the path node adjacent to the path end node of the target path in the target path, determine the target data source corresponding to the third group of path nodes adjacent to the target data source corresponding to the second group of path nodes in the path direction of the target path.

[0094] In this embodiment of the present application, it is assumed that the target data source corresponding to the third group of path nodes is the same as the target data source corresponding to the first group of path nodes, and path node I is a path node adjacent to path end node J. Since the second group of path nodes does not include path node I, the data analysis device continues to perform step 214. The target data source corresponding to the third group of path nodes is the first target data source.

[0095] Step 215: Obtain the fourth node data of the third group of path nodes from the target data source corresponding to the third group of path nodes, and repeat this process until the target task of each path node is executed based on the third sub-execution result and the fifth sub-data corresponding to each path node in the fifth node data, and the execution result of the target path is obtained.

[0096] Among them, the fifth node data is obtained from the target data source corresponding to the fourth group of path nodes including the path nodes adjacent to the path end node in the target path, and the third sub-execution result is obtained by executing the corresponding task on the data in the previous target data source adjacent to the target data source corresponding to the fourth group of path nodes.

[0097] In this embodiment of the present application, since path node I is adjacent to the destination node in the target path, the fourth reference node is the third reference node, and the fifth node number is the fourth node data. Consequently, the third sub-execution result is obtained, and the third sub-execution result is used as the execution result for the path. Accordingly, the fourth node data of path node I is obtained from the first target data source, and the second sub-execution result and the fourth node data of path node I are analyzed based on the target task corresponding to path node I to obtain the execution result for the target path E→F→G→H→I→J.

[0098] Step 216: Determine the fifth target data source to which the common endpoint node of the k paths in the target directed acyclic graph belongs.

[0099] Step 217: Acquire the sixth node data of the path endpoint node from the fifth target data source.

[0100] Step 218: Based on the sixth node data and the execution results of the k paths, a target analysis result for target intelligent analysis is obtained.

[0101] In other words, steps 205 to 218 implement the following: mainly analyzing the dependency relationship of the directed acyclic graph, analyzing the computing tasks in the direction of the directed acyclic graph, caching the computing tasks of the data source of each path node first, until a computing task that needs to cross the data source is encountered in the computing task chain, that is, in each target path direction, the previously cached computing tasks belonging to the same data source can be executed on a single node at one time, and parallel computing processing is performed on different computing task chains, that is, multiple execution engines process them simultaneously, aggregate them after processing, and finally return the execution results. It should be noted that one execution engine can be used to implement each computing task chain.

[0102] Based on the above embodiments, the present application provides an application scenario of a data analysis method, such as Figure 3 The target directed acyclic graph obtained by the method of the embodiment of the present application is shown. In the target directed acyclic graph, two data sources are included: data source 1 and data source 2, which are composed of two different paths, namely path 31 and path 32. Among them, path 31 starts from the path starting point node 1 and includes: path node 1, path node 2, path node 3, path node 5, path node 6 and path node 8 in sequence. Path 32 starts from the path starting point node 4 and path node 8 and follows the direction of the arrow to represent the dependency relationship between tasks. Among them, path node 1, path node 2, path node 3 and path node 4 belong to data source 1, and path node 5, path node 6, path node 7 and path node 8 belong to data source 2. Among them:

[0103] For path 31, step 1, obtain the data corresponding to path node 1, path node 2, and path node 3 from data source 1, and calculate the data corresponding to path node 1, path node 2, and path node 3 to obtain the first sub-execution result corresponding to path node 1, path node 2, and path node 3; step 2, obtain the data corresponding to path node 6 from the second data source 2; step 3, calculate the first sub-execution result corresponding to path node 1, path node 2, path node 3 and the data corresponding to path node 6 to obtain the execution result of path 31.

[0104] For path 32, step 4 obtains the data corresponding to path node 4 from data source 1 and performs calculations on the data corresponding to path node 4 to obtain the first sub-execution result corresponding to path node 4. Step 5 obtains the data corresponding to path node 7 from second data source 2. Step 6 calculates the first sub-execution result corresponding to path node 4 and the data corresponding to path node 7 to obtain the execution result of path 32. It should be noted that steps 1-3 and steps 4-6 are executed in parallel and there is no logical order in which they are executed.

[0105] Step 7: Based on path node 8, the execution results of path 31 and path 32 are summarized and aggregated to obtain the target analysis results.

[0106] It should be noted that, for the description of the same steps and contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0107] A data analysis method provided by an embodiment of the present application determines at least one data source to be analyzed used in a target intelligent analysis process, generates a target directed acyclic graph corresponding to at least one data source to be analyzed based on the at least one data source to be analyzed, and then, starting from the path starting point node of the target path in the target directed acyclic graph, grouping the path nodes in the target path according to the path direction of the target path based on the relationship that the path nodes belong to the same data source and are adjacent, obtains at least one group of path nodes of the target path, obtains node data of each group of path nodes corresponding to the target data source according to the path direction of the target path and analyzes the data, determines the execution result of the target path, and summarizes the execution results of k paths to obtain the target analysis result for the target intelligent analysis. In this way, when performing cross-data source query and analysis, a corresponding target directed acyclic graph is generated for at least one data source to be analyzed. Then, the path nodes in each target path in the target directed acyclic graph are grouped to obtain at least one group of path nodes for the target path. The node data of each group of path nodes is obtained from the target data source corresponding to the group of path nodes and analyzed to obtain the execution result of each target path. Finally, the execution results of k paths are summarized to obtain the target analysis result for the target intelligent analysis. This implements a technical solution for simultaneously extracting the node data of adjacent nodes belonging to the same target data source from the target data source, solving the current problem of high network IO overhead, reducing network IO overhead, and effectively improving the analysis efficiency of BI analysis. An AST is generated through SQL, and the target directed acyclic graph is obtained from the obtained AST for business intelligence analysis. This implements a unified SAL query engine as the core, while simultaneously obtaining data from different path nodes from the same data source for analysis. This improves the execution efficiency of the SQL engine, reduces the response time of BI analysis tool queries, and achieves richer data analysis capabilities.

[0108] Based on the above embodiments, the embodiments of the present application provide a data analysis device, which can be applied to Figures 1-2 In the data analysis method provided in the corresponding embodiment, refer to Figure 4 As shown, the data analysis device 4 may include: a determination unit 41, a generation unit 42, a processing unit 43 and a summary unit 44, wherein:

[0109] A determination unit 41 is configured to determine at least one data source to be analyzed used in a target intelligent analysis process;

[0110] A generating unit 42 is configured to generate a target directed acyclic graph corresponding to at least one data source to be analyzed based on at least one data source to be analyzed; wherein the target directed acyclic graph includes k paths, where k is an integer greater than or equal to 1;

[0111] The processing unit 43 is configured to group the path nodes in the target path in the target directed acyclic graph, starting from a path starting point node of the target path, according to the path direction of the target path, based on the relationship that the path nodes belong to the same data source and are adjacent, to obtain at least one group of path nodes of the target path; wherein the target path is used to represent each path in the k paths;

[0112] The processing unit 43 is further configured to obtain, from the target data source corresponding to each group of path nodes of the target path, node data corresponding to each group of path nodes according to the path direction of the target path, and analyze the node data to determine the execution result of the target path; wherein the target data source corresponding to each group of path nodes of the target path belongs to at least one data source to be analyzed;

[0113] The summarizing unit 44 is used to summarize the execution results of the k paths to obtain a target analysis result for target intelligent analysis.

[0114] In other embodiments of the present application, the generation unit 42 includes: a first generation module and a second generation module; wherein:

[0115] A first generating module is configured to generate an abstract syntax tree (AST) based on at least one data source to be analyzed;

[0116] The second generation module is used to generate a target directed acyclic graph based on the AST.

[0117] In other embodiments of the present application, the first generation module is used to implement the following steps:

[0118] Determining reference dimension parameters and reference metric parameters for each of at least one data source to be analyzed;

[0119] Based on the reference dimension parameters and reference metric parameters of each data source to be analyzed, the data in each data source to be analyzed is classified and packaged to obtain packaged data of each data source to be analyzed;

[0120] Obtaining target processing parameters; wherein the target processing parameters include target dimension parameters and / or target metric parameters;

[0121] Based on the target processing parameters, the encapsulated data of at least one data source to be analyzed is processed to obtain an AST.

[0122] In other embodiments of the present application, the first generation module is used to implement the step of processing the encapsulated data of at least one data source to be analyzed based on the target processing parameters to obtain the AST, which is specifically implemented by the following steps:

[0123] Generate SQL query statements based on the target processing parameters and preset SQL statement rules;

[0124] Using open source SQL parsing tools, based on SQL query statements, the encapsulated data of at least one data source to be analyzed is parsed to obtain an AST.

[0125] In other embodiments of the present application, the second generation module is specifically used to implement the following steps:

[0126] Verify the grammatical structure object in each node of AST;

[0127] If the grammatical structure object in each node of the AST passes the verification, the same subtrees in the AST are merged to obtain the target directed acyclic graph.

[0128] In other embodiments of the present application, the processing unit 43 includes: a first acquisition module and a first processing module; wherein:

[0129] a first acquisition module configured to acquire first node data of a group of path nodes from a target data source corresponding to the group of path nodes if the target path includes a group of path nodes; wherein the group of path nodes includes path nodes in the target path excluding a path endpoint node;

[0130] The first processing module is used to execute the target task of each path node in a group of path nodes in accordance with the path direction of the target path and based on the first sub-data corresponding to each path node in the first node data, to obtain the execution result of the target path.

[0131] In other embodiments of the present application, the processing unit 43 further includes: a first determination module; wherein:

[0132] a first determining module configured to determine, if the target path includes at least two groups of path nodes, a target data source corresponding to a first group of path nodes including a path starting point node of the target path according to a path direction of the target path;

[0133] The first acquisition module is further configured to acquire the second node data of the first group of path nodes from the target data source corresponding to the first group of path nodes;

[0134] The first processing module is further configured to execute the target task corresponding to each path node in the first group of path nodes based on the second sub-data corresponding to each path node in the second node data according to the path direction of the target path, and obtain a first sub-execution result;

[0135] The first determining module is further configured to determine, in a path direction of the target path, target data sources corresponding to a second group of path nodes adjacent to the target data sources corresponding to the first group of path nodes;

[0136] The first acquisition module is further configured to acquire the third node data of the second group of path nodes from the target data source corresponding to the second group of path nodes;

[0137] The first processing module is further configured to execute the target task corresponding to each path node in the second group of path nodes according to the path direction of the target path based on the first sub-execution result and the third sub-data corresponding to each path node in the third node data, to obtain a second sub-execution result;

[0138] The first determination module is further configured to determine that the execution result of the target path is the second sub-execution result if the second group of path nodes includes a path node in the target path that is adjacent to the path end node of the target path.

[0139] In other embodiments of the present application, the processing unit 43 further includes: a repeating module; wherein:

[0140] The first determining module is further configured to determine, in a path direction of the target path, target data sources corresponding to third group of path nodes adjacent to target data sources corresponding to the second group of path nodes if the second group of path nodes includes at least one path node other than a path node adjacent to a path end node of the target path in the target path;

[0141] A repetition module is used to obtain the fourth node data of the third group of path nodes from the target data source corresponding to the third group of path nodes, and repeat this process until the target task corresponding to each path node is executed based on the third sub-execution result and the fifth sub-data corresponding to each path node in the fifth node data, to obtain the execution result of the target path; wherein the fifth node data is obtained from the target data source corresponding to the fourth group of path nodes including the path nodes adjacent to the path end node in the target path, for the fourth group of path nodes, and the third sub-execution result is obtained by executing the corresponding task on the data in the previous target data source adjacent to the target data source corresponding to the fourth group of path nodes.

[0142] In other embodiments of the present application, the summarizing unit 44 includes: a second determining module, a second acquiring module, and a second processing module; wherein:

[0143] A second determining module is used to determine a fifth target data source to which a common path endpoint node of k paths in the target directed acyclic graph belongs;

[0144] A second acquisition module is used to acquire sixth node data of the path endpoint node from a fifth target data source;

[0145] The second processing module is used to obtain a target analysis result for target intelligent analysis based on the sixth node data and the execution results of the k paths.

[0146] It should be noted that the specific implementation process of the steps performed by the data analysis device in this embodiment can refer to Figures 1-2 The implementation process of the data analysis method provided in the corresponding embodiment will not be repeated here.

[0147] An embodiment of the present application provides a data analysis device, which determines at least one data source to be analyzed used in a target intelligent analysis process, generates a target directed acyclic graph corresponding to at least one data source to be analyzed based on the at least one data source to be analyzed, and then, starting from the path starting point node of the target path in the target directed acyclic graph, groups the path nodes in the target path according to the path direction of the target path based on the relationship that the path nodes belong to the same data source and are adjacent to each other, thereby obtaining at least one group of path nodes of the target path; from the target data source corresponding to each group of path nodes of the target path, obtains the node data of each corresponding group of path nodes according to the path direction of the target path and analyzes the data; determines the execution result of the target path; and summarizes the execution results of k paths to obtain the target analysis result for the target intelligent analysis. In this way, when performing cross-data source query and analysis, a corresponding target directed acyclic graph is generated for at least one data source to be analyzed. Then, the path nodes in each target path in the target directed acyclic graph are grouped to obtain at least one group of path nodes for the target path. The node data of each group of path nodes is obtained from the target data source corresponding to the group of path nodes and analyzed to obtain the execution result of each target path. Finally, the execution results of k paths are summarized to obtain the target analysis result for the target intelligent analysis. This implements a technical solution for simultaneously extracting the node data of adjacent nodes belonging to the same target data source from the target data source, solving the current problem of high network IO overhead, reducing network IO overhead, and effectively improving the analysis efficiency of BI analysis. An AST is generated through SQL, and the target directed acyclic graph is obtained from the obtained AST for business intelligence analysis. This implements a unified SAL query engine as the core, while simultaneously obtaining data from different path nodes from the same data source for analysis. This improves the execution efficiency of the SQL engine, reduces the response time of BI analysis tool queries, and achieves richer data analysis capabilities.

[0148] Based on the above embodiments, the embodiments of the present application provide a data analysis device, which can be applied to Figures 1-2 In the data analysis method provided in the corresponding embodiment, refer to Figure 5 As shown, the data analysis device 5 may include: a processor 51, a memory 52 and a communication bus 53, wherein:

[0149] A communication bus 53 is used to implement communication between the processor 51 and the memory 52;

[0150] Processor 51 is used to execute the data analysis program stored in memory 52 to achieve Figures 1-2 The implementation process of the data analysis method provided in the corresponding embodiment will not be repeated here.

[0151] Based on the above embodiments, the embodiments of the present application provide a computer-readable storage medium, referred to as a storage medium, which stores one or more programs, which can be executed by one or more processors to implement the reference Figures 1-2 The implementation process of the data analysis method provided in the corresponding embodiment will not be repeated here.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0156] The above description is merely a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application.

Claims

1. A data analysis method, characterized in that: The method comprises: Determine at least one data source to be analyzed used in the target intelligent analysis process; Based on the at least one data source to be analyzed, generating a target directed acyclic graph corresponding to the at least one data source to be analyzed; wherein the target directed acyclic graph includes k paths, where k is an integer greater than or equal to 1; Starting from a path starting point node of a target path in the target directed acyclic graph, grouping the path nodes in the target path according to the path direction of the target path based on the relationship that the path nodes belong to the same data source and are adjacent to each other, thereby obtaining at least one group of path nodes of the target path; wherein the target path is used to represent each of the k paths; and the grouping method is to divide the path nodes in the target path that belong to the same data source and are adjacent to each other into a group of path nodes; Obtaining, from a target data source corresponding to each group of path nodes of the target path, node data of each group of path nodes corresponding thereto according to the path direction of the target path and performing analysis to determine an execution result of the target path; wherein the target data source corresponding to each group of path nodes of the target path belongs to the at least one data source to be analyzed; obtaining, from a target data source corresponding to each group of path nodes of the target path, node data of each group of path nodes corresponding thereto according to the path direction of the target path and performing analysis, comprises: obtaining, from the target data source corresponding to each group of path nodes of the target path, node data of all path nodes of the group of path nodes at once, and analyzing the node data of each path node corresponding thereto according to the path direction of the group of path nodes in the target path; The execution results of the k paths are summarized to obtain a target analysis result for target intelligent analysis.

2. The method according to claim 1, characterized in that The generating, based on the at least one data source to be analyzed, a target directed acyclic graph corresponding to the at least one data source to be analyzed includes: Generate an abstract syntax tree (AST) based on the at least one data source to be analyzed; Based on the AST, the target directed acyclic graph is generated.

3. The method according to claim 2, characterized in that Generating an abstract syntax tree (AST) based on the at least one data source to be analyzed includes: Determining a reference dimension parameter and a reference metric parameter of each data source to be analyzed in the at least one data source to be analyzed; Based on the reference dimension parameters and the reference metric parameters of each of the data sources to be analyzed, classify and encapsulate the data in each of the data sources to be analyzed to obtain encapsulated data of each of the data sources to be analyzed; Obtaining target processing parameters; wherein the target processing parameters include target dimension parameters and / or target metric parameters; Based on the target processing parameters, the encapsulated data of the at least one data source to be analyzed is processed to obtain the AST.

4. The method according to claim 3, characterized in that The step of processing the encapsulated data of the at least one data source to be analyzed based on the target processing parameter to obtain the AST includes: Generate an SQL query statement based on the target processing parameters according to the preset SQL statement rules; The encapsulated data of the at least one data source to be analyzed is parsed based on the SQL query statement using an open source SQL parsing tool to obtain the AST.

5. The method according to claim 2, characterized in that Generating the target directed acyclic graph based on the AST includes: Verify the grammatical structure object in each node of the AST; If the grammatical structure object in each node of the AST passes the verification, the same subtrees in the AST are merged to obtain the target directed acyclic graph.

6. The method according to claim 1, characterized in that The step of obtaining, from a target data source corresponding to each group of path nodes of the target path according to the path direction of the target path, node data of each group of path nodes corresponding to the target path and analyzing the data to determine the execution result of the target path includes: If the target path includes a set of path nodes, obtaining first node data of the set of path nodes from a target data source corresponding to the set of path nodes; wherein the set of path nodes includes path nodes in the target path excluding the path endpoint node; According to the path direction of the target path, the target task of each path node in the group of path nodes is executed in sequence based on the first sub-data corresponding to each path node in the first node data to obtain the execution result of the target path.

7. The method according to claim 1, characterized in that The step of obtaining, from a target data source corresponding to each group of path nodes of the target path according to the path direction of the target path, node data of each group of path nodes corresponding to the target path is obtained and analyzed to determine the execution result of the target path, further comprising: If the target path includes at least two groups of path nodes, determining, according to the path direction of the target path, a target data source corresponding to a first group of path nodes including a path starting point node of the target path; Obtaining second node data of the first group of path nodes from a target data source corresponding to the first group of path nodes; executing the target task corresponding to each path node in the first group of path nodes based on the second sub-data corresponding to each path node in the second node data according to the path direction of the target path, to obtain a first sub-execution result; Determining, in the path direction of the target path, target data sources corresponding to a second group of path nodes adjacent to the target data sources corresponding to the first group of path nodes; Obtaining third node data of the second group of path nodes from a target data source corresponding to the second group of path nodes; executing the target task corresponding to each path node in the second group of path nodes according to the path direction of the target path based on the first sub-execution result and the third sub-data corresponding to each path node in the third node data, to obtain a second sub-execution result; If the second group of path nodes includes path nodes in the target path that are adjacent to the path end node of the target path, the execution result of the target path is determined to be the second sub-execution result.

8. The method according to claim 7, characterized in that The method further comprises: If the second group of path nodes includes at least one path node other than a path node adjacent to a path end node of the target path in the target path, determining, in the path direction of the target path, a target data source corresponding to a third group of path nodes adjacent to the target data source corresponding to the second group of path nodes; The fourth node data of the third group of path nodes is obtained from the target data source corresponding to the third group of path nodes, and this process is repeated until the target task corresponding to each path node is executed based on the third sub-execution result and the fifth sub-data corresponding to each path node in the fifth node data, to obtain the execution result of the target path; wherein the fifth node data is obtained from the target data source corresponding to the fourth group of path nodes including the path nodes adjacent to the path end node in the target path, for the fourth group of path nodes, and the third sub-execution result is obtained by executing the corresponding task on the data in the previous target data source adjacent to the target data source corresponding to the fourth group of path nodes.

9. The method according to any one of claims 2 to 8, characterized in that: Summarizing the execution results of the k paths to obtain target analysis results for target intelligent analysis includes: Determine a fifth target data source to which a common path endpoint node of the k target paths in the target directed acyclic graph belongs; Acquire sixth node data of the path endpoint node from the fifth target data source; Based on the sixth node data and the execution results of the k paths, the target analysis result for the target intelligent analysis is obtained.

10. A data analysis device, characterized in that: The device comprises: a determination unit, a generation unit, a processing unit and a summarization unit; wherein: The determining unit is configured to determine at least one data source to be analyzed used in the target intelligent analysis process; The generating unit is configured to generate a target directed acyclic graph corresponding to the at least one data source to be analyzed based on the at least one data source to be analyzed; wherein the target directed acyclic graph includes k paths, where k is an integer greater than or equal to 1; The processing unit is configured to group path nodes in the target path in the target directed acyclic graph, starting from a path starting point node of the target path, according to a path direction of the target path, based on relationships that the path nodes belong to the same data source and are adjacent, to obtain at least one group of path nodes in the target path; wherein the target path is used to represent each of the k paths; and the grouping method is to divide the path nodes in the target path that belong to the same data source and are adjacent into a group of path nodes; The processing unit is further configured to obtain, from a target data source corresponding to each group of path nodes in the target path, node data corresponding to each group of path nodes according to the path direction of the target path, and analyze the node data to determine an execution result of the target path; wherein the target data source corresponding to each group of path nodes in the target path belongs to the at least one data source to be analyzed; and is further configured to obtain, from the target data source corresponding to each group of path nodes in the target path, node data of all path nodes in the group of path nodes at once, and analyze the node data of each path node corresponding to the group of path nodes according to the path direction of the group of path nodes in the target path; The summarizing unit is used to summarize the execution results of the k paths to obtain a target analysis result for target intelligent analysis.

11. A data analysis device, characterized in that: The device includes a memory, a processor, and a communication bus; wherein: The memory is used to store executable instructions; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute the data analysis program stored in the memory to implement the steps of the data analysis method according to any one of claims 1 to 9.

12. A storage medium, characterized in that: The storage medium stores a data analysis program, which, when executed by a processor, implements the steps of the data analysis method according to any one of claims 1 to 9.

Images