Real-time query method and system for multi-source heterogeneous data

Abstract

The invention discloses a real-time query method and system for multi-source heterogeneous data. The method comprises the steps that a visual data analysis system receives a query request of a user and sends the query request to an analysis engine; the analysis engine constructs a query plan according to the query request; the analysis engine queries a first multi-dimensional data set stored by the pre-calculation engine according to the query plan to obtain a second multi-dimensional data set and sends the second multi-dimensional data set to the execution calculation engine; and / or, the task scheduling engine generates a query task queue based on the query plan, and the execution calculation engine queries the multi-source heterogeneous data according to the query task queue to obtain a third multi-dimensional data set; and the execution engine generates a query result according to the second multi-dimensional data set and / or the third multi-dimensional data set, and returns the query result to the visual data analysis system, so that the user can perform visual analysis on the query result. By means of the scheme provided by the embodiment, efficient real-time query and analysis can be conducted on multi-source heterogeneous mass data, and the visual data analysis requirement of a user is met.

Description

Technical Field

[0001] This article relates to the field of big data technology, and in particular to a real-time query method and system for multi-source heterogeneous data. Background Technology

[0002] With the continuous advancement of big data technology, data storage methods are becoming increasingly diversified. Common data sources include relational databases, non-relational databases, and various data warehouses. However, traditional multi-source heterogeneous data query technologies suffer from low query efficiency and fail to consider the data querying issues of non-relational databases. Especially when handling complex query tasks initiated by the front end, they cannot guarantee the real-time performance and efficiency of queries, making it difficult to meet the needs of real-time querying of multi-source heterogeneous data in massive data environments.

[0003] Furthermore, traditional query techniques require system administrators to pre-configure the data model or cube to be queried in the system. After the system completes the query, the results are returned to the user for visualization and analysis. This technique cannot flexibly adjust the queried cube according to the user's real-time needs and requires the participation of technical personnel for pre-configuration, making it difficult to adapt to the current analytical needs of complex, multi-source, heterogeneous data. Therefore, how to efficiently query this multi-source, heterogeneous data has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method and system for real-time querying of multi-source heterogeneous data, which can efficiently query and analyze massive amounts of multi-source heterogeneous data in real time, meeting users' needs for visual data analysis.

[0005] One embodiment of this application provides a real-time query method for multi-source heterogeneous data, including: The visualization data analysis system receives user query requests and sends the query requests to the parsing engine; The parsing engine parses the query request and constructs a query plan based on the parsing results; The parsing engine queries the first multidimensional dataset stored in the pre-computation engine according to the query plan to obtain the second multidimensional dataset and sends it to the execution computing engine; and / or, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third multidimensional dataset. The execution computing engine generates query results based on the second multidimensional dataset and / or the third multidimensional dataset, and returns them to the visualization data analysis system; wherein, the first multidimensional dataset is a multidimensional dataset generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data; The visualization data analysis system receives the query results so that users can perform visualization analysis on the query results.

[0006] In one exemplary embodiment, the parsing engine queries the first multidimensional dataset stored in the pre-computation engine according to the query plan to obtain a second multidimensional dataset and sends it to the execution computation engine; and / or, the task scheduling engine generates a query task queue based on the query plan, and the execution computation engine queries the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset, including: If all metadata items in the query plan match all metadata items in the first cube, then the parsing engine queries the first cube stored in the pre-computation engine according to the query plan to obtain the second cube and sends it to the execution computation engine; or, If a metadata item in the query plan partially matches a metadata item in the first cube, the parsing engine queries the first cube stored in the pre-computation engine based on the matching metadata item in the query plan to obtain a second cube and sends it to the execution computation engine. The remaining unmatched metadata items in the query plan are sent to the task scheduling engine. The task scheduling engine generates a query task queue based on the remaining unmatched metadata items in the query plan. The execution computation engine then queries the multi-source heterogeneous data according to the query task queue to obtain a third cube; or... If the metadata items in the query plan do not match the metadata items in the first multidimensional dataset, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third multidimensional dataset. The metadata items include dimensions, metrics, and filtering conditions.

[0007] In one exemplary embodiment, the execution computing engine queries the multi-source heterogeneous data according to the query task queue, including: If the target data of the query task in the query task queue is in the batch cache database, the execution computing engine queries the target data from the batch cache database; The process of establishing the batch cache database includes: For each database table in the multi-source heterogeneous data, perform the following operations: Get the query data volume of the database table in the first time period, every second time period, and sort the query data volume in ascending or descending order to generate the query data volume trend curve of the database table; Calculate the area of ​​the first triangle formed between the maximum value point, the minimum value point and the first intermediate point on the trend curve of the query data volume. When the area of ​​the first triangle is the maximum, the first intermediate point is determined as an inflection point. Calculate the area of ​​the second triangle formed between the maximum value point, the first inflection point and the second intermediate point on the trend curve of the query data volume. When the area of ​​the second triangle is the maximum, the second intermediate point is determined as the second inflection point. The amount of query data corresponding to the second inflection point is determined as the amount of cached data in the database table, and the data in the database table with the highest query frequency whose data volume does not exceed the amount of cached data is cached in the batch cache database; Wherein, the first intermediate point is any point between the maximum and minimum points on the trend curve of the query data volume; the second intermediate point is any point between the maximum point and a first inflection point on the trend curve of the query data volume.

[0008] In one exemplary embodiment, the execution computing engine queries the multi-source heterogeneous data according to the query task queue, further comprising: For query tasks of NoSQL databases in the query task queue, the execution computing engine converts the query task into a NoSQL query task and then queries the NoSQL database.

[0009] In one exemplary embodiment, the method further includes: The message queue engine receives real-time collected data; The execution computing engine acquires real-time collected data from the message queue engine and processes it in real time, storing the processing results in the stream database; The visualization data analysis system obtains processed real-time collected data from the streaming database in real time, so that users can perform visualization analysis.

[0010] In one exemplary embodiment, after the visualization data analysis system receives a user's query request and before sending the query request to the parsing engine, the method further includes: If the query request is hit in the query plan cache, the query plan for the query request is directly obtained from the query plan cache and sent to the pre-computation engine and / or the task scheduling engine for processing; The query plan cache stores the mapping relationship between historical query requests and corresponding query plans.

[0011] In one exemplary embodiment, the task scheduling engine generates a query task queue based on the query plan, including: The task scheduling engine generates query tasks based on the query plan, and constructs a task topology graph based on the type of the generated query tasks and the query plan. The resource manager allocates resources to tasks on each node in the task topology graph based on the task topology graph. The task scheduling engine places the query tasks of the corresponding nodes into the query task queue according to the execution order of the task topology graph. The types of query tasks include remote query tasks and multi-source data query tasks; the remote query task is a query task that can be completed independently at a single data source; the multi-source data query task is a query task that merges the data returned by multiple remote query tasks; the multi-source data query task is executed using a multi-node distributed query method; in the task topology graph, each query task is represented as a node, and the dependencies between query tasks are represented using directed edges.

[0012] In one exemplary embodiment, the first cube is generated according to the following method: The pre-computation engine exhaustively enumerates and pre-computes all possible aggregate query datasets based on the aggregate query operation of the query request, constructs the first multidimensional dataset based on all the exhaustively enumerated aggregate query datasets, and stores it in the pre-computation engine; and configures the first multidimensional dataset to be fully constructed or incrementally constructed based on the data volume of the first multidimensional dataset.

[0013] In one exemplary embodiment, the parsing engine parses the query request and constructs a query plan based on the parsing results, including: The parsing engine parses and splits the query request to obtain the AST syntax tree of the query request; for each query operation node in the syntax tree, the query type of the node is determined, and the actual database name and actual database table name of the data to be queried by the node are determined based on the mapping relationship table stored in the virtual data model; the query plan is constructed according to the query type, actual database name and database table name of all query nodes in the syntax tree. The query types include single-data source queries or multi-data source queries; the virtual data model stores a mapping table between virtual database names, virtual database table names, and actual database names and actual database table names; the query plan includes metadata items.

[0014] Another embodiment of this application provides a real-time query system for multi-source heterogeneous data, including: A visualization data analysis system is used to receive user query requests and send the query requests to the parsing engine; it also receives query results returned by the execution calculation engine so that users can perform visualization analysis on the query results. The parsing engine is used to parse the query request and construct a query plan based on the parsing results; according to the query plan, it queries the first multidimensional dataset stored in the pre-computation engine to obtain a second multidimensional dataset and sends it to the execution engine; wherein, the first multidimensional dataset is a multidimensional dataset generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data; A task scheduling engine is used to generate a query task queue based on the query plan; The execution computing engine is used to query the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset; generate query results based on the second multidimensional dataset and / or the third multidimensional dataset, and return them to the visualization data analysis system.

[0015] The technical solutions provided in this application can efficiently query and analyze massive amounts of heterogeneous data from multiple sources in real time, meeting users' needs for visualized data analysis.

[0016] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the embodiments described in the description and the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0018] Figure 1 A flowchart illustrating a real-time query method for multi-source heterogeneous data provided in this application embodiment; Figure 2 A schematic diagram of the data processing flow for the real-time query method for multi-source heterogeneous data provided in the embodiments of this application; Figure 3 A schematic diagram of the data processing flow for the real-time query method for multi-source heterogeneous data provided in the embodiments of this application; Figure 4 This is a schematic diagram of the cache data volume algorithm provided in the embodiments of this application; Figure 5 This is a schematic diagram of the AST syntax tree provided in an embodiment of this application. Detailed Implementation

[0019] This application describes several embodiments, but these descriptions are exemplary and not limiting, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0020] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application can also be combined with any conventional features or elements to form unique inventive solutions. Any feature or element of any embodiment can also be combined with features or elements from other inventive solutions to form another unique inventive solution. Therefore, it should be understood that any feature shown and / or discussed in this application can be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes can be made within the scope of the appended claims.

[0021] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0022] The following describes in detail the real-time query method and system for multi-source heterogeneous data provided in this application, one by one, in the embodiments.

[0023] Figure 1 This illustration shows a flowchart of a real-time query method for multi-source heterogeneous data according to an embodiment of this application. Figure 1 As shown, the method includes: Step 100: The visualization data analysis system receives the user's query request and sends the query request to the parsing engine; Step 101: The parsing engine parses the query request and constructs a query plan based on the parsing results; Step 102: The parsing engine queries the first multidimensional dataset stored in the pre-computation engine according to the query plan to obtain the second multidimensional dataset and sends it to the execution computing engine; and / or, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third multidimensional dataset; Step 103: The execution calculation engine generates query results based on the second multidimensional dataset and / or the third multidimensional dataset, and returns them to the visualization data analysis system; Step 104: The visualization data analysis system receives the query results so that the user can perform visualization analysis on the query results.

[0024] In this embodiment, the first multidimensional dataset is a multidimensional dataset generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data.

[0025] In this embodiment, as Figure 2 As shown, a multi-source heterogeneous data source can include various types of data sources, such as relational databases, non-relational databases, data lakes / warehouses, etc., to store the multi-source heterogeneous data that users need to analyze.

[0026] A centralized data management warehouse provides a unified query interface for accessing and manipulating multi-source heterogeneous data sources by offering a standard SQL syntax and building a virtual data model.

[0027] The distributed real-time data query and analysis framework provides a big data processing framework that can include a parsing engine, a pre-computation engine, a task scheduling engine, and an execution computing engine. It is used to perform real-time retrieval, querying, and analysis of data from multiple heterogeneous data sources and generate one or more multidimensional datasets as query results.

[0028] The visualization data analysis system is used to receive user query requests and send them to the parsing engine for querying through a centralized data management warehouse, or to obtain query results so that users can perform self-service visualization data analysis on the query results.

[0029] like Figure 2 As shown, the specific steps of the real-time query method for multi-source heterogeneous data include: Step 200: The user initiates a query request on the visualization data analysis system and sends the query request to the parsing engine. This query request is typically an SQL query. Step 201: The parsing engine parses the SQL statement in the query request and constructs a query plan based on the parsing results; Step 202: The parsing engine queries the first cube stored in the pre-computation engine according to the query plan, obtains the second cube, and sends it to the execution computation engine, and / or; Step 203: The task scheduling engine splits, allocates, and schedules the query plan into tasks, generating a query task queue; Step 204: The execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third multidimensional dataset, and generates query results based on the second and / or third multidimensional datasets, and returns them to the visualization data analysis system; Step 205: The visualization data analysis system receives the query results, enabling users to perform self-service visualization data analysis on the system based on the returned cube query results.

[0030] In this embodiment, a user initiates a real-time query operation targeting a multi-source heterogeneous data source. After the parsing engine generates a query plan based on the query request, it can directly query the first cube stored in the pre-computation engine to obtain the second cube. Alternatively, the task scheduling engine generates a query task queue based on the query plan, and the execution computation engine queries the multi-source heterogeneous data according to the query task queue to obtain the third cube. After obtaining the second and third cubes, the execution computation engine merges the second and third cubes to obtain the query results. Since the first cube is generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data, it already contains the pre-processed results of data frequently queried and calculated by the user, and its data volume is much smaller than the total data volume of the multi-source heterogeneous data source. Therefore, there is no need to separately plan task queues and execution resources for this part of the query plan to perform a full query from the massive amount of multi-source heterogeneous data; instead, the second cube can be quickly and directly queried from the first cube. This query method can significantly reduce the computing resources and I / O overhead required for real-time queries, thereby improving the efficiency of real-time query response for user query requests.

[0031] In this embodiment, the visual data analysis system provides users with an intuitive and easy-to-use interface for real-time querying of multi-source heterogeneous data. The visual data analysis system may include a real-time data management module, a real-time data model module, and a visualization module.

[0032] The real-time data management module manages the query results generated in real time, which can be one or more multidimensional datasets. It also includes an SQL editing area for users to input query requests for multi-source heterogeneous data. The query results managed by this module are generated in real time based on the SQL query statements written by the user in the editing area. The module also allows users to set automatic update times for the query results, such as updating them every 1 minute or 3 minutes.

[0033] In this embodiment, the real-time data model module enables users to perform secondary editing and statistical analysis of query results, or to construct new data analysis models based on the query results. For example, users can integrate relevant data according to business needs or across business domains to perform secondary modeling of the query results, forming new multidimensional datasets available for self-service visualization analysis. As another example, if a user executes multiple real-time queries and obtains query results for multiple multidimensional datasets, and if some or all of these multidimensional datasets share common dimensions, the user can use the real-time data model module to perform secondary correlation modeling of the query results for multiple multidimensional datasets based on these common dimensions, constructing a new multidimensional dataset. Furthermore, the real-time data management module also supports obtaining corresponding data models by selecting an API. For example, by selecting a pre-configured API, the real-time data model module can directly generate and display a dataset for the user based on the data model obtained from the API.

[0034] In this embodiment, through the visualization module, users can select the query results of the multidimensional dataset to be analyzed using the visualization dashboard, generate appropriate visualization charts by dragging and dropping dimensions and numerical fields, and build a visualization analysis dashboard to present data analysis results and trends in a clear and intuitive way, and independently complete visualization data analysis.

[0035] In the real-time query method for multi-source heterogeneous data provided in this embodiment, users can easily initiate real-time query requests for multi-source heterogeneous data by writing query conditions through a visual data analysis system. They can flexibly adjust the multidimensional dataset to be queried according to real-time needs and perform real-time data visualization analysis of the query results in the visual data analysis system interface. Therefore, it does not require the participation of professional technicians to pre-configure the multidimensional dataset to be queried in the system. Moreover, since this embodiment uses a pre-computation engine to pre-build the first multidimensional dataset, the parsing engine can directly query the first multidimensional dataset stored in the pre-computation engine after generating the query plan, thereby obtaining the second multidimensional dataset to generate query results, greatly improving query efficiency. Therefore, the technical solution provided in this embodiment can meet users' needs for self-service, efficient real-time querying and visual data analysis of complex, multi-source heterogeneous, and massive data sources at both the technical and strategic levels.

[0036] In one exemplary embodiment, the parsing engine queries the first multidimensional dataset stored in the pre-computation engine according to the query plan to obtain a second multidimensional dataset and sends it to the execution computation engine; and / or, the task scheduling engine generates a query task queue based on the query plan, and the execution computation engine queries the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset, including: If all metadata items in the query plan match all metadata items in the first cube, then the parsing engine queries the first cube stored in the pre-computation engine according to the query plan to obtain the second cube and sends it to the execution computation engine; or, If a metadata item in the query plan partially matches a metadata item in the first cube, the parsing engine queries the first cube stored in the pre-computation engine based on the matching metadata item in the query plan to obtain a second cube and sends it to the execution computation engine. The remaining unmatched metadata items in the query plan are sent to the task scheduling engine. The task scheduling engine generates a query task queue based on the remaining unmatched metadata items in the query plan. The execution computation engine then queries the multi-source heterogeneous data according to the query task queue to obtain a third cube; or... If the metadata items in the query plan do not match the metadata items in the first multidimensional dataset, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third multidimensional dataset. The metadata items include dimensions, metrics, and filtering conditions.

[0037] In this embodiment, a query plan refers to the detailed execution steps generated in a database management system to execute a query statement. The query plan describes how the database accesses and processes data to satisfy the query request. Before executing the query plan, the metadata items in the query plan, such as dimensions, metrics, and filtering conditions, are first compared with the query plan or metadata items of the first multidimensional dataset stored in the pre-computation engine. Specifically, this includes three cases: complete match, partial match, and no match.

[0038] If all metadata items in the query plan match the metadata items in the first cube, the parsing engine queries the first cube stored in the pre-computation engine according to the query plan, and the pre-computation engine directly returns the second cube and sends it to the execution calculation engine.

[0039] If a metadata item in the query plan partially matches a metadata item in the first cube—for example, if the dimension and metric information of a subquery in the query plan corresponds to a metadata item in the first cube stored in the pre-computation engine—then for the matching metadata item in the query plan, the parsing engine queries the first cube stored in the pre-computation engine according to the query plan. The pre-computation engine directly returns the second cube and sends it to the execution engine, without needing to recalculate based on the data stored in the multi-source heterogeneous data source. The remaining unmatched metadata items in the query plan are sent to the task scheduling engine. The task scheduling engine generates a query task queue based on the remaining unmatched metadata items in the query plan, and then the execution calculation engine queries the multi-source heterogeneous data according to the query task queue to obtain the third cube.

[0040] If the metadata items in the query plan do not match the metadata items in the first cube, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third cube.

[0041] Finally, when some or all of the metadata items in the query plan match the metadata items of the first cube in the pre-computation engine, the matching metadata items in the query plan need to be converted into a query plan for the first cube. The pre-computation engine then queries the first cube according to the query plan for the first cube to generate the second cube.

[0042] In this embodiment, before executing the query plan on the multi-source heterogeneous data, the parsing engine first matches the metadata items of the first multidimensional dataset that have been pre-computed and stored in the pre-computation engine. If a complete or partial match is satisfied, the parsing engine can directly query the matching metadata items from the first multidimensional dataset stored in the pre-computation engine to obtain the second multidimensional dataset. There is no need to separately plan task queues and execution resources for the matching metadata items to query the multi-source heterogeneous data. Thus, the pre-computation engine greatly saves the computation and IO overhead of subsequent queries and improves the efficiency of queries.

[0043] In one exemplary embodiment, the execution computing engine queries the multi-source heterogeneous data according to the query task queue, including: If the target data of the query task in the query task queue is in the batch cache database, the execution computing engine queries the target data from the batch cache database; The process of establishing the batch cache database includes: For each database table in the multi-source heterogeneous data, perform the following operations: Get the query data volume of the database table in the first time period, every second time period, and sort the query data volume in ascending or descending order to generate the query data volume trend curve of the database table; Calculate the area of ​​the first triangle formed between the maximum value point, the minimum value point and the first intermediate point on the trend curve of the query data volume. When the area of ​​the first triangle is the maximum, the first intermediate point is determined as an inflection point. Calculate the area of ​​the second triangle formed between the maximum value point, the first inflection point and the second intermediate point on the trend curve of the query data volume. When the area of ​​the second triangle is the maximum, the second intermediate point is determined as the second inflection point. The amount of query data corresponding to the second inflection point is determined as the amount of cached data in the database table, and the data in the database table with the highest query frequency whose data volume does not exceed the amount of cached data is cached in the batch cache database; Wherein, the first intermediate point is any point between the maximum and minimum points on the trend curve of the query data volume; the second intermediate point is any point between the maximum point and a first inflection point on the trend curve of the query data volume.

[0044] In this embodiment, as Figure 3As shown, a batch cache database can be used to cache database tables from multiple heterogeneous data sources, and to pre-cache frequently queried tables. The batch cache database can be implemented using an in-memory database. If the target data for a query task in the query queue is located in the batch cache database, the target data is retrieved from the batch cache database first, thus avoiding the large amount of disk I / O operations generated by querying from multiple heterogeneous data sources and improving data query performance.

[0045] This embodiment provides a batch data caching mechanism that caches data in batches for frequently queried database tables based on the actual data query volume from multiple heterogeneous data sources. For each database table, the cached data volume is determined using a quadratic inflection point calculation method based on the query data volume within a first time period.

[0046] The process of establishing a batch cache database includes: For each database table in a multi-source heterogeneous dataset, the cached data size for that database table is determined using the following method: Retrieve the query data volume of this database table within the first time period, at every second time period, and sort it in descending or ascending order to form a query data volume trend curve, such as... Figure 4 As shown, connect the maximum and minimum points on the curve, and then connect the first intermediate point between the maximum and minimum points to the maximum and minimum points respectively, forming a triangular region. When the area of ​​the triangle formed by the first intermediate point, the maximum point, and the minimum point is the largest, the first intermediate point on the curve is determined as the first inflection point.

[0047] The area of ​​a triangle can be calculated using Heron's formula: Where S is the area of ​​the triangle; p is the semi-perimeter, i.e. (a+b+c) / 2; and a, b, and c are the lengths of the three sides of the triangle, respectively.

[0048] After obtaining the first inflection point, set it as the new minimum point, while keeping the maximum point unchanged. Determine a second intermediate point between the first inflection point and the maximum point on the curve. When the area of ​​the triangle formed by the second intermediate point, the maximum point, and the first inflection point is maximized, determine the second intermediate point as the secondary inflection point. Set the amount of query data corresponding to the secondary inflection point as the maximum amount of data that should be cached in this database table.

[0049] The cached data volume for each database table determined by the above algorithm can cover the range of high-frequency query data in most daily scenarios. Moreover, the maximum cached data volume is less than the highest query volume for this database table in a recent period. This can reduce the storage pressure on the batch cache database, thereby avoiding frequent updates to cached data or the need to bypass the batch cache database and obtain data from the source database of a large number of heterogeneous multi-source data sources due to cache misses, which would cause additional IO overhead.

[0050] For any multi-source heterogeneous data involving cross-database operations or high-frequency joins, materialized views can be pre-created for the relevant database tables. Materialized views can be used to optimize complex queries. When a query task is executed, the multi-source heterogeneous data source checks if a pre-created materialized view exists to complete the query request. If so, the query can be quickly completed based on the materialized view, thereby improving query efficiency. Furthermore, materialized views can be configured for automatic refresh to maintain data consistency. The batch caching mechanism for materialized views can be configured to be consistent with the batch caching mechanism for database tables described above.

[0051] Furthermore, when the amount of data in a query task exceeds the amount of data stored in the batch cache database, or when the query time range is greater than the time range of the data in the current batch cache database, the cache data volume of the corresponding database table can be updated based on the historical query data volume within the most recent first time period at the current moment, using the algorithm described above. At the same time, the data currently stored in the batch cache database is updated incrementally in real time. This ensures that the data required for the query can be quickly retrieved from the batch cache database during the query, avoiding queries in databases with multiple heterogeneous data sources, reducing the number of database accesses, and improving query efficiency.

[0052] The technical solution provided in this application provides that, for query tasks performed on multi-source heterogeneous data sources, pre-caching is performed on frequently queried database tables. During a query, cached data is preferentially retrieved from the batch cache database, further improving data query efficiency. Furthermore, this embodiment employs a separate algorithm design for the cached data volume of each database table and can dynamically update the cached data for each table. This ensures that the cached data volume of database tables with large query volumes can cover the needs of most query scenarios, while the cached data volume of database tables with low query volumes gradually decreases. Cached data in database tables that have not been used for a long time is cleared from the batch cache database after a period of time, thus meeting query requirements in most scenarios while minimizing the size of the batch cache database. Additionally, this embodiment can pre-generate materialized views of data from common cross-database and join operations. When a query task can hit a materialized view, it can directly query relevant data from that materialized view, greatly simplifying the query operation and improving query performance. Using the above-mentioned batch caching mechanism to cache materialized views of database tables further improves query efficiency.

[0053] In one exemplary embodiment, the execution computing engine queries the multi-source heterogeneous data according to the query task queue, further comprising: For query tasks of NoSQL databases in the query task queue, the execution computing engine converts the query task into a NoSQL query task and then queries the NoSQL database.

[0054] In this embodiment, since the SQL query syntax differs between different types of relational databases, a unified standard SQL syntax can be pre-established in a centralized data management warehouse for joint analysis and querying of various databases from multiple heterogeneous data sources. Users only need to write query statements according to the standard SQL syntax, without needing to worry about the syntax differences between individual databases. The execution engine will parse the query task and rewrite it into SQL query statements that can be correctly executed in the actual databases of each type before executing the corresponding query.

[0055] If there are NoSQL database query tasks in the query task queue, the execution computing engine can convert the query tasks into NoSQL query tasks by calling components or middleware, and then query the NoSQL database to obtain the data from the non-relational database.

[0056] In one exemplary embodiment, the method further includes: The message queue engine receives real-time collected data; The execution computing engine acquires real-time collected data from the message queue engine and processes it in real time, storing the processing results in the stream database; The visualization data analysis system obtains processed real-time collected data from the streaming database in real time, so that users can perform visualization analysis.

[0057] The technical solution provided in this embodiment can further integrate real-time streaming data processing and query functions on the basis of batch data query, forming a data processing mode that integrates streaming and batch processing. In this embodiment, a streaming database can also be set up in the centralized data management warehouse. The streaming database is responsible for storing the data stream of processing results of real-time collected data that has high requirements for the real-time performance of data statistics. For example, in the process of business processing, certain special events, alarms, monitoring data, etc., have high requirements for statistical real-time performance. This part of the data can be collected in real time, and the real-time collected data can be synchronized and temporarily stored in real time through a message queue engine. Then, the data in the message queue is consumed and processed in real time through an execution computing engine, and the processing results are synchronously stored in the streaming database.

[0058] like Figure 3 As shown, some real-time acquired data from the raw data is directly stored in the message queue engine. The execution engine retrieves the real-time acquired data from the message queue engine and processes it in real time. The processing results are stored in the streaming database. The visualization data analysis system is configured with corresponding API interfaces to directly access the data stream stored in the streaming database. The streaming database and the visualization analysis system can communicate in real time based on a bidirectional communication protocol. Once the data in the streaming database is updated, the new data can be pushed to the visualization data analysis system in real time via the API, allowing users to independently perform highly real-time visualization analysis of the streaming data.

[0059] The specific steps of the streaming data query scheme are as follows: Step 300: The message queue engine consumes a portion of the real-time collected data from the raw data in real time; Step 301: Execute the computing engine to process the data in the message queue in real time; Step 302: The processed data stream is further stored in the stream database; Step 303: The visualization data analysis system obtains data streams from the streaming database in real time through the corresponding API interface.

[0060] For example, the message queue engine can be configured to collect and consume real-time data streams related to events, logs, alarms, or performance relevant to the current business. Assuming the log event data to be collected in real-time includes fields such as "log name, log type, event type, event level, source IP, source port, destination IP, destination port, occurrence time, occurrence count," etc., the execution calculation engine can further consume and process this data stream from the message queue engine. This includes processing such as field simplification and aggregating identical events by occurrence count. The processing result includes "log name, event type, event level, source IP, destination IP, occurrence time, and occurrence count." The processed data stream is automatically synchronized to the stream database in real-time. The visualization data analysis system is configured with an API to retrieve data from the stream database. The system also includes a user interface that supports field configuration and selection to meet users' needs for independent statistical analysis. Users can select this API as the data source in the dashboard interface of the visualization data analysis system and construct visual charts by dragging and dropping dimensions and numerical fields to independently complete visualization data analysis. Since data updates in the streaming database are pushed to the API interface of the visualization data analysis system in real time, real-time updates of streaming data can also be achieved in the dashboard of the visualization data analysis system.

[0061] In one exemplary embodiment, after the visualization data analysis system receives a user's query request and before sending the query request to the parsing engine, the method further includes: If the query request is hit in the query plan cache, the query plan for the query request is directly obtained from the query plan cache and sent to the pre-computation engine and / or the task scheduling engine for processing; The query plan cache stores the mapping relationship between historical query requests and corresponding query plans.

[0062] In this embodiment, after receiving a user's query request, the visualization data analysis system can check the query plan cache to see if the current query request already exists before sending it to the parsing engine. The query plan cache is used to pre-cachise optimized query plans for frequently queried SQL query requests. When the same SQL query request is initiated again, there is no need to re-parse and optimize the query request; the existing query plan can be directly retrieved from the query plan cache.

[0063] Since the SQL query requests submitted by users in the visualization data analysis system usually need to be polled and updated periodically, by pre-parsed and optimized the user's query requests, and storing the mapping relationship between the query requests and optimized query plans in the query plan cache, when the user's query request hits the query plan cache, the cached query plan can be used directly, which significantly reduces the parsing and optimization time required by the system when processing the same query request, thereby improving the overall query performance.

[0064] In one exemplary embodiment, the task scheduling engine generates a query task queue based on the query plan, including: The task scheduling engine generates query tasks based on the query plan, and constructs a task topology graph based on the type of the generated query tasks and the query plan. The resource manager allocates resources to tasks on each node in the task topology graph based on the task topology graph. The task scheduling engine places the query tasks of the corresponding nodes into the query task queue according to the execution order of the task topology graph. The types of query tasks include remote query tasks and multi-source data query tasks; the remote query task is a query task that can be completed independently at a single data source; the multi-source data query task is a query task that merges the data returned by multiple remote query tasks; the multi-source data query task is executed using a multi-node distributed query method; in the task topology graph, each query task is represented as a node, and the dependencies between query tasks are represented using directed edges.

[0065] In this embodiment, the task scheduling engine generates query tasks by splitting and allocating tasks based on the query plan. These query tasks are divided into remote query tasks and multi-source data query tasks. Remote query tasks can be directly sent to the database of the multi-source heterogeneous data source to complete the query and filtering. Multi-source data query tasks, on the other hand, take the data returned by the remote query tasks and perform intersection or merging through multi-node distributed queries. This decentralized execution method effectively reduces the amount of full data transmission during the multi-source heterogeneous data source query processing, saving significant IO overhead and improving data processing efficiency.

[0066] Taking JOIN operations as an example, for JOIN operations across data sources, the query process before the JOIN operation can be performed on the data source side (i.e., executing remote query tasks). If the JOIN operation is performed within a single data source, then the JOIN operation can also be completed on the data source side. Afterwards, based on the data returned by the remote query, subsequent JOIN operations or merging operations are performed on the multi-source data query task side through multi-node distributed queries.

[0067] The task scheduling engine, based on the classification of query tasks and query plans, represents each query task as a node, and each query task as an independent unit of work. Dependencies between tasks are represented using directed edges, generating a task topology graph. The execution order of tasks is determined by sorting the task topology graph, and each query task is scheduled step-by-step according to this order. Query task priorities are established, and synchronous or asynchronous execution is determined based on the dependencies between query tasks. For tasks without dependencies, parallel execution can be used to improve resource utilization.

[0068] Once the execution order of the query tasks is determined, the resource manager allocates resources to the nodes. For example, it allocates single-machine or cluster node resources for remote query tasks deployed with multiple heterogeneous data sources, and allocates node resources within the deployment cluster of the distributed real-time data query and analysis framework for multi-source data query tasks. Finally, the query tasks are placed into the query task queue.

[0069] In one exemplary embodiment, the first cube is generated according to the following method: The pre-computation engine exhaustively enumerates and pre-computes all possible aggregate query datasets based on the aggregate query operation of the query request, constructs the first multidimensional dataset based on all the exhaustively enumerated aggregate query datasets, and stores it in the pre-computation engine; and configures the first multidimensional dataset to be fully constructed or incrementally constructed based on the data volume of the first multidimensional dataset.

[0070] In this embodiment, the core concept of the pre-computation engine is to pre-compute and store the data that the user may need to query. In this way, during the actual query process, the user can directly access the multidimensional dataset stored in the pre-computation engine to perform the query, avoiding the use of the execution engine to query data from multiple heterogeneous data sources. This can save a lot of query time, thereby reducing the amount of computation and time delay required when the user actually performs the query.

[0071] First, the pre-computation engine exhaustively searches for all possible combinations of aggregated query data based on the aggregated query operations in the user's submitted query request, and performs pre-computation to form the first multidimensional dataset stored in the pre-computation engine. For example, if the user submits a query request for a dataset of (country, city, district, population), then the pre-computation engine will further exhaustively search for multidimensional datasets of (country, city, population), (country, district, population), (city, district, population), (city, population), (country, population), (district, population) to cover all possible query subsets and ranges, maximizing the time savings for subsequent users performing similar queries.

[0072] Furthermore, the pre-computation engine supports both full and incremental construction of the first cube. For small-volume first cubes or those frequently requiring full table updates, full construction reduces redundant computations and lowers maintenance complexity in production environments. For large-volume first cubes, such as one containing two years of historical data, recalculating the past two years' worth of data for newly added data becomes wasteful if the cube needs daily updates. In such cases, incremental construction can be considered, enabling real-time multidimensional analysis of the first cube with sub-second latency.

[0073] Finally, since the first cube generated by the pre-computation engine gradually occupies a large amount of storage space, it is advisable to periodically clear cubes that have not been used or have been inactive for a long time by setting an expiration time. This approach preserves frequently used cubes while reducing server storage pressure by periodically releasing pre-computation engine storage space.

[0074] In one exemplary embodiment, the parsing engine parses the query request and constructs a query plan based on the parsing results, including: The parsing engine parses and splits the query request to obtain the AST syntax tree of the query request; for each query operation node in the syntax tree, the query type of the node is determined, and the actual database name and actual database table name of the data to be queried by the node are determined based on the mapping relationship table stored in the virtual data model; the query plan is constructed according to the query type, actual database name and database table name of all query nodes in the syntax tree. The query types include single-data source queries or multi-data source queries; the virtual data model stores a mapping table between virtual database names, virtual database table names, and actual database names and actual database table names; the query plan includes metadata items.

[0075] In this embodiment, the virtual data model provides a unified access name for databases and tables in multi-source heterogeneous data sources. The virtual data model only defines how to uniformly retrieve data from various data sources, without synchronizing or storing data from the real data sources. For users, it is convenient to use a set of aliases for databases and tables to establish logical connections with various actual databases and tables in multi-source heterogeneous data sources to access source data, without needing to care about the distribution of the underlying real data sources.

[0076] In a virtual data model, logical connections between relational database tables and non-relational databases can be established simultaneously. External tables can also be defined to link data in non-relational databases, allowing users to query data in non-relational databases using SQL statements. Virtual data models prevent conflicts between tables with the same name from multiple data sources, avoid the need for physical migration of large and complex data, reduce resource consumption, and allow access to various data sources with simple logical connections and configurations.

[0077] A unified data query interface, based on a virtual data model, provides a consistent data query interface for multi-source heterogeneous data sources. The query operations performed by users based on the virtual database names and virtual database table names provided by the virtual data model will ultimately be assigned to the corresponding actual databases and database tables for execution.

[0078] In this embodiment, the unified data query interface also provides a standard SQL syntax for the visual data analysis system to perform SQL queries. This effectively simplifies query operations across multiple data sources without requiring the learning of a specific query language for each data source. Furthermore, even if new data sources are added later, creating a multi-source heterogeneous data source environment, only new mapping relationships need to be added to the virtual data model, without requiring extensive modifications to the existing application. The execution engine at the lower level parses and rewrites the standard SQL query requests into statements that can be correctly executed in the actual database.

[0079] The parsing engine is primarily responsible for recursively parsing and splitting the SQL statements in the user's query request to form an Abstract Syntax Tree (AST). For each node in the AST, it determines whether the query is from a single or multiple data sources. Based on the mapping relationship of the virtual data model, it identifies the actual database name and table name to be queried for each node. Finally, based on the query type, actual database name, and table name of all query nodes in the syntax tree, a specific query plan is constructed.

[0080] The following example illustrates the processing of a query request across multiple data sources: The SQL statement for the query request entered by the user is as follows: SELECT s1.nid, s1.name, s2.sgroup FROM ( SELECT c.nid FROM my_A.test_1 AS c JOIN ( SELECT a.nid, b.sname FROM pg_A.test_1 AS a JOIN oc.test_1 AS b ON a.nid = b.nid WHERE a.nid > 100 AND b.nid < 1000 AS t ON c.nid = t.nid WHERE c.nid > 200 ) AS s1 JOIN ( SELECT a.nid, b.sgroup FROM hive_A.test 1 AS a JOIN ck_A.test1 AS bON a.nid = b.nid WHERE a.nid > 100 AND b.nid < 1000) AS s2 ON s1.nid = s2.nid In a MySQL database with multiple heterogeneous data sources, the `test` database table under the database named `mysql_A` can be set as an alias for a `test1` database table under the database named `my_A`. When users perform queries in the self-service visual data analysis system, they can use standard SQL syntax to query the `test` database table by alias (e.g., `SELECT *FROM my_A.test1`).

[0081] After receiving this query request, the parsing engine will recursively parse the query request and generate a result such as... Figure 5 The AST syntax tree shown is used to find the actual names corresponding to the virtual database names and virtual database table names in each node according to the mapping relationship in the virtual data model, so as to ensure that the corresponding query tasks can be executed normally.

[0082] In the above query requests, database table query operations take precedence over cross-data source JOIN and merge operations, and both database table query operations and cross-data source JOIN and merge operations can be executed in parallel. The query operations on the database tables pg_A.test1, oc_A.test1, my_A.test1, hive_A.test1, and ck_A.test1 (in this example, primarily represented by WHERE conditions) can all be completed on the data source side. Depending on the specific database deployment method, they can be run on a single machine or multiple nodes, which is a single-data source query. Cross-source JOIN and merge operations, however, require the result data to be sent to the distributed query framework after the data source query is completed, and then completed through a multi-node distributed query approach, which can further improve query efficiency.

[0083] Once all query tasks are completed, the execution calculation engine sends the multidimensional dataset query results to the visualization data analysis system for users to perform visualization analysis.

[0084] Another embodiment of this application provides a real-time query system for multi-source heterogeneous data, including: A visualization data analysis system is used to receive user query requests and send the query requests to the parsing engine; it also receives query results returned by the execution calculation engine so that users can perform visualization analysis on the query results. The parsing engine is used to parse the query request and construct a query plan based on the parsing results; according to the query plan, it queries the first multidimensional dataset stored in the pre-computation engine to obtain a second multidimensional dataset and sends it to the execution engine; wherein, the first multidimensional dataset is a multidimensional dataset generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data; A task scheduling engine is used to generate a query task queue based on the query plan; The execution computing engine is used to query the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset; generate query results based on the second multidimensional dataset and / or the third multidimensional dataset, and return them to the visualization data analysis system.

[0085] The specific working process of the real-time query system for multi-source heterogeneous data in this embodiment can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0086] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A real-time query method for multi-source heterogeneous data, characterized in that, include: The visualization data analysis system receives user query requests and sends the query requests to the parsing engine; The parsing engine parses the query request and constructs a query plan based on the parsing results; The parsing engine queries the first multidimensional dataset stored in the pre-computation engine according to the query plan, obtains the second multidimensional dataset, and sends it to the execution computation engine. And / or, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset; The execution computing engine generates query results based on the second multidimensional dataset and / or the third multidimensional dataset, and returns them to the visualization data analysis system; wherein, the first multidimensional dataset is a multidimensional dataset generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data; The visualization data analysis system receives the query results so that users can perform visualization analysis on the query results.

2. The real-time query method for multi-source heterogeneous data according to claim 1, characterized in that, The parsing engine queries the first multidimensional dataset stored in the pre-computation engine according to the query plan, obtains the second multidimensional dataset, and sends it to the execution computation engine. And / or, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset, including: If all metadata items in the query plan match all metadata items in the first cube, then the parsing engine queries the first cube stored in the pre-computation engine according to the query plan to obtain the second cube and sends it to the execution computation engine; or, If a metadata item in the query plan partially matches a metadata item in the first cube, the parsing engine queries the first cube stored in the pre-computation engine based on the matching metadata item in the query plan to obtain a second cube and sends it to the execution computation engine. The remaining unmatched metadata items in the query plan are sent to the task scheduling engine. The task scheduling engine generates a query task queue based on the remaining unmatched metadata items in the query plan. The execution computation engine then queries the multi-source heterogeneous data according to the query task queue to obtain a third cube; or... If the metadata items in the query plan do not match the metadata items in the first multidimensional dataset, the task scheduling engine generates a query task queue based on the query plan, and the execution computing engine queries the multi-source heterogeneous data according to the query task queue to obtain the third multidimensional dataset. The metadata items include dimensions, metrics, and filtering conditions.

3. The real-time query method for multi-source heterogeneous data according to claim 1 or 2, characterized in that, The execution computing engine queries the multi-source heterogeneous data according to the query task queue, including: If the target data of the query task in the query task queue is in the batch cache database, the execution computing engine queries the target data from the batch cache database; The process of establishing the batch cache database includes: For each database table in the multi-source heterogeneous data, perform the following operations: Get the query data volume of the database table in the first time period, every second time period, and sort the query data volume in ascending or descending order to generate the query data volume trend curve of the database table; Calculate the area of ​​the first triangle formed between the maximum value point, the minimum value point and the first intermediate point on the trend curve of the query data volume. When the area of ​​the first triangle is the maximum, the first intermediate point is determined as an inflection point. Calculate the area of ​​the second triangle formed between the maximum value point, the first inflection point and the second intermediate point on the trend curve of the query data volume. When the area of ​​the second triangle is the maximum, the second intermediate point is determined as the second inflection point. The amount of query data corresponding to the second inflection point is determined as the amount of cached data in the database table, and the data in the database table with the highest query frequency whose data volume does not exceed the amount of cached data is cached in the batch cache database; Wherein, the first intermediate point is any point between the maximum and minimum points on the trend curve of the query data volume; the second intermediate point is any point between the maximum point and a first inflection point on the trend curve of the query data volume.

4. The real-time query method for multi-source heterogeneous data according to claim 3, characterized in that, The execution computing engine queries the multi-source heterogeneous data according to the query task queue, and also includes: For query tasks of NoSQL databases in the query task queue, the execution computing engine converts the query task into a NoSQL query task and then queries the NoSQL database.

5. The real-time query method for multi-source heterogeneous data according to claim 1, characterized in that, The method further includes: The message queue engine receives real-time collected data; The execution computing engine acquires real-time collected data from the message queue engine and processes it in real time, storing the processing results in the stream database; The visualization data analysis system obtains processed real-time collected data from the streaming database in real time, so that users can perform visualization analysis.

6. The real-time query method for multi-source heterogeneous data according to claim 3, characterized in that, After the visualization data analysis system receives a user's query request, and before sending the query request to the parsing engine, the method further includes: If the query request is hit in the query plan cache, the query plan for the query request is directly obtained from the query plan cache and sent to the pre-computation engine and / or the task scheduling engine for processing; The query plan cache stores the mapping relationship between historical query requests and corresponding query plans.

7. The real-time query method for multi-source heterogeneous data according to claim 1, characterized in that, The task scheduling engine generates a query task queue based on the query plan, including: The task scheduling engine generates query tasks based on the query plan, and constructs a task topology graph based on the type of the generated query tasks and the query plan. The resource manager allocates resources to tasks on each node in the task topology graph based on the task topology graph. The task scheduling engine places the query tasks of the corresponding nodes into the query task queue according to the execution order of the task topology graph. The types of query tasks include remote query tasks and multi-source data query tasks; the remote query task is a query task that can be completed independently at a single data source; the multi-source data query task is a query task that merges the data returned by multiple remote query tasks; the multi-source data query task is executed using a multi-node distributed query method; in the task topology graph, each query task is represented as a node, and the dependencies between query tasks are represented using directed edges.

8. The real-time query method for multi-source heterogeneous data according to claim 1, characterized in that, The first cube was generated according to the following method: The pre-computation engine exhaustively enumerates and pre-computes all possible aggregate query datasets based on the aggregate query operation of the query request, constructs the first multidimensional dataset based on all the exhaustively enumerated aggregate query datasets, and stores it in the pre-computation engine; Furthermore, the first cube is configured to be built in full or incrementally based on the amount of data in the first cube.

9. The real-time query method for multi-source heterogeneous data according to claim 1, characterized in that, The parsing engine parses the query request and constructs a query plan based on the parsing results, including: The parsing engine parses and splits the query request to obtain the AST syntax tree of the query request; for each query operation node in the syntax tree, the query type of the node is determined, and the actual database name and actual database table name of the data to be queried by the node are determined based on the mapping relationship table stored in the virtual data model; the query plan is constructed according to the query type, actual database name and database table name of all query nodes in the syntax tree. The query types include single-data source queries or multi-data source queries; the virtual data model stores a mapping table between virtual database names, virtual database table names, and actual database names and actual database table names; the query plan includes metadata items.

10. A real-time query system for multi-source heterogeneous data, characterized in that, include: A visual data analysis system is used to receive user query requests and send the query requests to a parsing engine; Receive the query results returned by the execution computing engine so that the user can perform visual analysis of the query results; The parsing engine is used to parse the query request and construct a query plan based on the parsing results; According to the query plan, the first multidimensional dataset stored in the pre-computation engine is queried to obtain the second multidimensional dataset and sent to the execution engine; wherein, the first multidimensional dataset is a multidimensional dataset generated by the pre-computation engine through pre-computation processing of the multi-source heterogeneous data; A task scheduling engine is used to generate a query task queue based on the query plan; The execution computing engine is used to query the multi-source heterogeneous data according to the query task queue to obtain a third multidimensional dataset; generate query results based on the second multidimensional dataset and / or the third multidimensional dataset, and return them to the visualization data analysis system.

Images