Data retrieval system of mixed storage and unified query layer based on AI cloud desktop

Abstract

The invention provides a mixed storage and unified query layer data retrieval system based on an AI cloud desktop, which belongs to the technical field of AI intelligent application and comprises a unified query access layer, an SQL optimizer, a vectorization execution engine, a mixed storage engine and a metadata management center. Column storage, inverted index and row storage are organically combined, services are provided for the outside through a unified SQL query layer, and intelligent routing and vectorization computing technologies are introduced. The method is characterized in that an optimal storage engine is automatically selected for execution according to a query mode, the problems that a single inverted index architecture is low in efficiency and high in resource consumption under the scenes of ad-hoc analysis, high-concurrency point search and large-scale aggregation are solved, and better comprehensive query performance, lower data storage cost and higher system expansibility than traditional Elasticsearch are achieved.

Description

Technical Field

[0001] This invention relates to the field of AI intelligent application technology, and in particular to a data retrieval system based on AI cloud desktop with hybrid storage and unified query layer. Background Technology

[0002] Elasticsearch (based on Apache Lucene) is one of the most popular full-text search engines, but its architecture has some inherent flaws:

[0003] High storage costs: In pursuit of retrieval speed, the original text needs to be converted into various data structures (inverted index, DocValues, row storage), resulting in severe data expansion and high storage costs.

[0004] Weak ad-hoc analysis capabilities: When performing aggregation and sorting operations on fields that are not pre-built in DocValues, the field values ​​need to be reconstructed from _source, resulting in huge performance overhead and even memory overflow.

[0005] Poor resource isolation: Complex aggregation queries and high-concurrency point lookup or retrieval queries can interfere with each other, easily exhausting cluster resources and leading to decreased stability.

[0006] Scalability bottleneck: Once the number of shards is set, it is difficult to modify, and reindexing is costly. Shards are the smallest units for data migration and load balancing. Shards that are too large can lead to hotspots, while shards that are too small can cause metadata explosion, affecting cluster stability.

[0007] Existing alternatives (such as using a hybrid architecture of "ClickHouse + Elasticsearch") can solve the problem to some extent, but they require maintaining two systems, resulting in complex architecture, data redundancy, and high operation and maintenance costs. Summary of the Invention

[0008] To address the above technical issues, this invention provides a data retrieval system based on AI cloud desktop with hybrid storage and a unified query layer.

[0009] The technical solution of this invention is:

[0010] A data retrieval system based on AI cloud desktop, featuring hybrid storage and a unified query layer, including...

[0011] The unified query access layer provides a standard SQL interface (compatible with the PostgreSQL protocol), reducing the learning and migration costs for users, while also providing a foundation for advanced functions such as complex multi-table joins and subqueries.

[0012] The query processing layer includes the SQL parser and optimizer, the intelligent query router, and the vectorized execution engine.

[0013] The hybrid storage engine integrates several storage engines and receives data from the intelligent query router.

[0014] The metadata management hub manages all data partitions, index information, and node status.

[0015] Furthermore,

[0016] It organically combines columnar storage, inverted indexes, and row-based storage, provides services to the outside world through a unified SQL query layer, and introduces intelligent routing and vectorized computing technology to automatically select the optimal storage engine for execution based on the query pattern.

[0017] Furthermore,

[0018] The intelligent query router automatically decomposes and routes queries to the optimal storage engine in the hybrid storage engine based on the characteristics of the SQL's WHERE conditions and GROUP BY fields.

[0019] Furthermore,

[0020] The vectorized execution engine performs all computational operations in a vectorized manner, utilizing CPU cache and SIMD instructions to improve computational efficiency.

[0021] SQL parsing and optimization: The SQL optimizer performs syntax parsing and rewriting of the query to generate a preliminary execution plan.

[0022] Furthermore,

[0023] The data partition has a fixed size and can be dynamically merged and split.

[0024] Furthermore,

[0025] The client submits queries using the standard SQL protocol.

[0026] Furthermore,

[0027] Intelligent routing analyzes query predicates based on information from the metadata hub:

[0028] If the query contains full-text search conditions such as MATCH, the subtask will be routed to the inverted index engine.

[0029] If the query mainly involves multi-column filtering and large-scale aggregation (GROUP BY), then it is routed to the columnar storage engine.

[0030] If the query is a primary key = query or an IN query with a small amount of data, it is routed to the row storage engine.

[0031] Furthermore,

[0032] After receiving the subtask, each storage engine uses a vectorized execution engine to perform data scanning, filtering, and computation.

[0033] The partial results returned by each storage engine are merged and sorted in the computation layer for final processing; the final results are then returned to the client.

[0034] The beneficial effects of this invention are

[0035] Aggregate query performance: 5-10 times faster in ad-hoc aggregation scenarios without pre-built indexes.

[0036] Resource utilization: Due to higher computing efficiency and no resource interference, the concurrent throughput that can be supported is increased by 3 times under the same hardware conditions.

[0037] Storage space: The same raw data occupies only 30% of the storage space of Elasticsearch.

[0038] Data timeliness: Data is available immediately after being written, without the waiting delay of refresh_interval in Elasticsearch. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] This invention proposes a data retrieval system based on AI cloud desktop, featuring hybrid storage and a unified query layer, including...

[0042] Unified SQL query layer: Provides a standard SQL interface (compatible with PostgreSQL protocol) to reduce user learning and migration costs, while providing a foundation for advanced functions such as complex multi-table joins and subqueries.

[0043] Hybrid Storage Engines and Intelligent Routing: The system integrates multiple storage engines and innovatively designs an intelligent query router. Based on the characteristics of SQL such as WHERE conditions and GROUP BY fields, it automatically decomposes queries and routes them to the optimal storage engine for execution.

[0044] Vectorized execution engine: Drawing on the advanced design of modern MPP databases, all computational operations (scanning, filtering, aggregation) are executed in a vectorized manner, making full use of CPU cache and SIMD instructions to significantly improve computational efficiency.

[0045] Dynamic and scalable metadata management: An independent metadata management hub is introduced to manage all data partitions, index information, and node status. Data partitions have a fixed size and can be dynamically merged and split, achieving true elastic scaling of storage and computing resources.

[0046] Query access: The client submits queries via the standard SQL protocol.

[0047] Parsing and Optimization: The SQL optimizer performs syntax parsing and rewriting of the query to generate a preliminary execution plan.

[0048] Intelligent routing: The intelligent query router is the key to this invention. It analyzes query predicates based on information from the metadata hub.

[0049] If the query contains full-text search conditions such as MATCH, the subtask will be routed to the inverted index engine.

[0050] If the query mainly involves multi-column filtering and large-scale aggregation (GROUP BY), then it is routed to the columnar storage engine.

[0051] If the query is a primary key = query or an IN query with a small amount of data, it is routed to the row storage engine.

[0052] Vectorized execution: After receiving the subtask, each storage engine uses the vectorized execution engine to perform data scanning, filtering and calculation, which greatly improves the processing capacity of a single node.

[0053] Result merging: The partial results returned by each storage engine are merged, sorted, and processed in the computation layer.

[0054] Result return: Returns the final result to the client.

[0055] This invention organically combines columnar storage, inverted indexes, and row-based storage, providing services through a unified SQL query layer. It solves the problems of low efficiency and high resource consumption of a single inverted index architecture in ad-hoc analysis, high-concurrency point queries, and large-scale aggregation scenarios.

[0056] The above description is merely a preferred embodiment of the present invention and is used only to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data retrieval system based on AI cloud desktop with hybrid storage and unified query layer, characterized in that, include A unified query access layer provides a standard SQL interface to the outside world; The query processing layer includes the SQL parser and optimizer, the intelligent query router, and the vectorized execution engine; Hybrid storage engine, which integrates several storage engines, receives data from intelligent query routers and decomposes it; The metadata management hub manages all data partitions, index information, and node status.

2. The system according to claim 1, characterized in that, It organically combines columnar storage, inverted indexes, and row-based storage, provides services to the outside world through a unified SQL query layer, and introduces intelligent routing and vectorized computing technology to automatically select the optimal storage engine for execution based on the query pattern.

3. The system according to claim 1, characterized in that, The intelligent query router automatically decomposes and routes queries to the optimal storage engine in the hybrid storage engine based on the characteristics of the SQL's WHERE conditions and GROUP BY fields.

4. The system according to claim 1, characterized in that, The vectorized execution engine performs all computational operations in a vectorized manner, utilizing CPU cache and SIMD instructions to improve computational efficiency.

5. The system according to claim 1, characterized in that, The data partition has a fixed size and can be dynamically merged and split.

6. The system according to claim 1, characterized in that, The client submits queries using the standard SQL protocol.

7. The system according to claim 1, characterized in that, SQL parsing and optimization: The SQL optimizer performs syntax parsing and rewriting of the query to generate a preliminary execution plan.

8. The system according to claim 1, characterized in that, Intelligent routing analyzes query predicates based on information from the metadata hub: If the query includes full-text search criteria, the subtask is routed to the inverted index engine; If the query involves multi-column filtering or large-scale aggregation, it will be routed to the columnar storage engine; If the query is a primary key = query or an IN query with a small amount of data, it is routed to the row storage engine.

9. The system according to claim 1 or 4, characterized in that, After receiving the subtask, each storage engine uses a vectorized execution engine to perform data scanning, filtering, and computation.

10. The system according to claim 9, characterized in that, The partial results returned by each storage engine are merged and sorted in the computation layer for final processing; the final results are then returned to the client.

Images